Status Codes and URL limited

theanti9 / PyCrawler

A python web crawler

212 stars 103 forks source link

Closed sjparsons closed 13 years ago

sjparsons commented 13 years ago

I added two features. 1) Logging of status codes. 2) A command-line argument for limiting searching to URLs that match the given regex.

Let me know your thoughts.

theanti9 commented 13 years ago

Thanks for contributing! It's much appreciated.

sjparsons commented 13 years ago

You're welcome! I've got another idea, but I'll post that separately.