search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

Improve connection handling #24

Open stewartmckee opened 10 years ago

stewartmckee commented 10 years ago

Seem to have issues with connections to redis sometimes under load, need to give ability to specify your own redis and and check handling of dropped connections.