search

stewartmckee / cobweb

Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
MIT License
227 stars 45 forks source link

I think the gem need to require json when using it standalone #14

Closed ghost closed 12 years ago

ghost commented 12 years ago

Hi,

when I was testing the gem today, and running the commands here :

statistics = CobwebCrawler.new(:cache => 600).crawl("http://www.pepsico.com") do |page|
  puts "Just crawled #{page[:url]} and got a status of #{page[:status_code]}."
end

got these errors:

!!!!!!!!!!!! ERROR !!!!!!!!!!!!!!!!
#<NoMethodError: undefined method `to_json' for {"text/html"=>1}:Hash>
[
    [0] "/var/lib/gems/1.9.1/gems/cobweb-0.0.68/lib/stats.rb:88:in `update_statistics'",
    [1] "/var/lib/gems/1.9.1/gems/cobweb-0.0.68/lib/cobweb_crawler.rb:85:in `block in crawl'"
]

So I think the gem need to require 'json' and modified the gemspec.

Thanks, MQ

stewartmckee commented 12 years ago

What version of ruby are you using, i've just tested with 1.9.3, clean rvm gemset with just installing cobweb and it seems to run ok. I'll pull in the change though, as it does require it.