search

OriginLive / SpideR

A web crawler engine that gathers data, sorts it and outputs it to file. The goal is for it to be fully customizable and extensible, with possibilities for scripting the behaviour of the spiders.
MIT License
2 stars 3 forks source link

POLITENESS FIX #8

Closed kingRodian closed 7 years ago

kingRodian commented 7 years ago

There was an error in the parsing of robots.txt which prohibited the bot from being polite. It is now fixed (I believe)