search

taganaka / polipus

Polipus: distributed and scalable web-crawler framework
MIT License
92 stars 32 forks source link

Fails when response["Set-Cookie"] is nil #36

Closed tmaier closed 10 years ago

tmaier commented 10 years ago 
TypeError: no implicit conversion of nil into String
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie/scanner.rb:20:in `initialize'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie/scanner.rb:20:in `initialize'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie.rb:281:in `new'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie.rb:281:in `block in parse'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie.rb:280:in `tap'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie.rb:280:in `parse'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie_jar.rb:191:in `parse'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/polipus-0.3.0/lib/polipus/http.rb:180:in `get_response'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/polipus-0.3.0/lib/polipus/http.rb:149:in `get'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/polipus-0.3.0/lib/polipus/http.rb:47:in `fetch_pages'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/polipus-0.3.0/lib/polipus.rb:183:in `block (3 levels) in takeover'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/redis-queue-0.0.4/lib/redis/queue.rb:56:in `block in process'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/redis-queue-0.0.4/lib/redis/queue.rb:54:in `loop'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/redis-queue-0.0.4/lib/redis/queue.rb:54:in `process'
/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/polipus-0.3.0/lib/polipus.rb:158:in `block (2 levels) in takeover'
taganaka commented 10 years ago

seems like it came out once we removed the StandardError catch all

taganaka commented 10 years ago

did you find and easy way to replicate it?

tmaier commented 10 years ago

Fell again over this one. I don't know how to test it

TypeError: no implicit conversion of nil into String
from /usr/local/opt/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/http-cookie-1.0.2/lib/http/cookie/scanner.rb:20:in `initialize'

I haven't found a proper way to test it, but here is a call which raises this error:

Polipus::HTTP.new(accept_cookies: true).fetch_page('http://www.bgebauer.de/')