search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

3309 wf fix riffle on crawl #43

Closed wflanagan closed 8 years ago

wflanagan commented 8 years ago

Fixes problems with Riffle and 301-based initial domains.

wflanagan commented 8 years ago

Sorry. meant do do this on my fork.. :-)