To reduce server load, could follow crawl-delay directions in robots.txt

Samita53 / ldspider

Automatically exported from code.google.com/p/ldspider

0 stars 0 forks source link

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago

It would be great if the bot could follow the Crawl-delay extension to the
robots.txt protocol to avoid overloading a server.

Original issue reported on code.google.com by ansell.p...@gmail.com on 16 Apr 2010 at 1:44

GoogleCodeExporter commented 9 years ago

workaround for now: reconfigure data source to limit access in general: 
http://code.google.com/p/ldspider/wiki/ServerConfig

Original comment by andr...@harth.org on 7 Jun 2010 at 5:55