search

sensiblecodeio / data-services-helpers

Python module containing classes and functions that The Sensible Code Company's Data Services often used
https://sensiblecode.io/
BSD 2-Clause "Simplified" License
4 stars 4 forks source link

Respect robots.txt by default #1

Open fawkesley opened 10 years ago

fawkesley commented 10 years ago

And probably provide a context manager with ignore_robots for rare occasions when we encounter a broken robots.txt file.

fawkesley commented 10 years ago

See robotparser in Python standard library: http://docs.python.org/2/library/robotparser.html