search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

Standalone Crawler gives error for redis #25

Closed Shehbaz closed 9 years ago

Shehbaz commented 10 years ago

When trying to use it as stand alone crawler without redis it gives error for redis connection.

If you run this:

crawler = CobwebCrawler.new(:cache => 600) statistics = crawler.crawl("http://www.pepsico.com")

You will get error Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)

While according to documentation, it should run without requiring redis in this case. Thanks

stewartmckee commented 10 years ago

Hi Shehbaz,

It does require redis, even for the standalone as it uses this to handle queues of crawled, queued urls and the statistics, i'll look at the documentation for this,

thanks, Stewart.

Shehbaz commented 10 years ago

Thanks for the quick response, here is the exact text:

CobwebCrawler

CobwebCrawler is the standalone crawling class. If you "don’t want to use redis" and just want to crawl the site within your ruby process, you can use this class.

Thanks again.

stewartmckee commented 10 years ago

Ok, I've had a look, Redis is required in the code, its the documentation thats wrong. I'll update the documentation, sorry for any confusion and thanks for pointing it out.

Its something I would like to have, ie not requiring redis for standalone version, so will look to add this back in the future,

thanks, Stewart.

stewartmckee commented 10 years ago

Think it was a typo redis > resque...

CobwebCrawler is the standalone crawling class. If you don’t want to use resque or sidekiq and just want to crawl the site within your ruby process, you can use this class.

Shehbaz commented 10 years ago

Thanks and if I can help in that let me know. Thanks

Shehbaz Bashir Ahmed | CEO Devbox Tel | Mob: +92-321-4392208 [image: WebsiteGB] http://www.devbx.com/ [image: email]shehbazbashir@devbx.com

On Tue, Jan 7, 2014 at 3:07 PM, Stewart McKee notifications@github.comwrote:

Ok, I've had a look, Redis is required in the code, its the documentation thats wrong. I'll update the documentation, sorry for any confusion and thanks for pointing it out.

Its something I would like to have, ie not requiring redis for standalone version, so will look to add this back in the future,

thanks, Stewart.

— Reply to this email directly or view it on GitHubhttps://github.com/stewartmckee/cobweb/issues/25#issuecomment-31725752 .

mur-wtag commented 8 years ago

sudo apt-get install redis-server would help you (yes, it's for linux).