stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

Redirect Limit causing crawl to stop #36

Open stewartmckee opened 9 years ago

stewartmckee commented 9 years ago

When the redirect limit is hit it kills the crawl. The RedirectError is thrown but doesn't seem to be trapped, it seems to be thrown for each subsequent call into the get method, which it shouldn't do because it should check that the redirect_limit has count down to 0.