Add referer and depth to Page if HTTP#fetch_pages has an error

taganaka / polipus

Polipus: distributed and scalable web-crawler framework

MIT License

92 stars 32 forks source link

Add referer and depth to Page if HTTP#fetch_pages has an error #23

Closed tmaier closed 10 years ago

tmaier commented 10 years ago

Might be necessary for #15

I would feel much better, if we would not rescue from StandardError but from the errors which are related to fetching a page (HTTP or connectivity error)

coveralls commented 10 years ago

Coverage remained the same when pulling 4495867f65f2a1b92d7ff4e3adabe0f5f0b0f648 on tmaier:http into 428ed675fa11e7375cd0e8bbfbf86fb1033c7e8e on taganaka:master.

taganaka commented 10 years ago

:+1: