ericholscher / django-crawler

A crawler using the Django Test Client
http://django-crawler.readthedocs.org
30 stars 10 forks source link

add '--exclude' option #6

Closed chrisv2 closed 9 years ago

chrisv2 commented 11 years ago

patch which adds a '--exclude' option, consisting of comma-separated list of patterns to exclude from crawl. excluding stuff like ^/admin can significantly speed up the crawl.

ericholscher commented 9 years ago

Closing this, but feel free to resend it with an updated patch. Sorry this took so long for me to get back to :(