search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
226 stars 45 forks source link

More crawl finished reliability work #8

Closed rojotek closed 11 years ago

rojotek commented 11 years ago

Stew,

Have updated the code after the merge, and also put in some extra tracking of things. I think we are in a much much better place with respect to the crawl_finished queue now.

Rob