search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

Suggestion: Compatibility with Sidekiq #17

Closed NebJ closed 10 years ago

NebJ commented 11 years ago

Hi @stewartmckee,

Great job for Cobweb, it's a very configurable & powerful tool!

I have a suggestion: Cobweb should be compatible with Sidekiq to enhance its performance.

What do you think?

stewartmckee commented 11 years ago

Totally agree... actually have a branch with it as an optional process engine, so defaults to resque, but you can switch to sidekiq. At the time cobweb wasn't threadsafe, but I think with some work i've been doing on multi-threading CobwebCrawler it may now be threadsafe, so can progress this further.

I'll update here when I have progressed this.

NebJ commented 11 years ago

Awesome! Let me know if I can do some work on cobweb ;)

stewartmckee commented 11 years ago

pushed up a branch for sidekiq (using_sidekiq), its not passing all specs yet though, something not right around the detection of the end of the crawl which needs a bit more work.

stewartmckee commented 10 years ago

This is now merged to master and ready to ship.