I received an email from a local newspaper last night. They noticed the scraper (it only took them five months!), and they've asked me to rate-limit the process. I've put a time.sleep(0.3) at the end of baseparser.py's grab_url() , which should keep them happy.
I'm opening this issue so people know about the problem.
I received an email from a local newspaper last night. They noticed the scraper (it only took them five months!), and they've asked me to rate-limit the process. I've put a
time.sleep(0.3)at the end of baseparser.py'sgrab_url(), which should keep them happy.I'm opening this issue so people know about the problem.