search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
226 stars 45 forks source link

Changes to redirect and crawl finished #10

Closed rojotek closed 11 years ago

rojotek commented 11 years ago

This commit includes two changes.

1) fixes the way that redirects happen -- properly updating the uri and the redirect through tracking code. 2) add a new value to the crawl finished job - showing which url was actually visited for the base url.