search

ndmitchell / hoogle

Haskell API search engine
http://hoogle.haskell.org/
Other
738 stars 134 forks source link

Don't log people who ask for robots.txt #121

Open ndmitchell opened 9 years ago

ndmitchell commented 9 years ago

A clear sign they are probably a robot. Store such a list of IP addresses and avoid them.