search

dakrone / itsy

A threaded web-spider written in Clojure
181 stars 30 forks source link

Politeness #5

Closed shriphani closed 10 years ago

shriphani commented 10 years ago

Crawler must obey robots.txt. User can still override this setting.

dakrone commented 10 years ago

Merged, thanks!