search

theanti9 / PyCrawler

A python web crawler
212 stars 104 forks source link

logging HTTP status codes #4

Closed sjparsons closed 13 years ago

sjparsons commented 13 years ago

I'm wondering what you think about logging the HTTP status codes on the URLs?

200 / 301 / 302 / 403 / 400 / 500 ?

http://www.w3.org/Protocols/HTTP/HTRESP.html

theanti9 commented 13 years ago

Ah yes, that's a good idea. I don't really have time to work on this project right now, so I encourage you to fork it and if you'd like, when you are done, we can merge the changes into here. Thank you for the idea, though.