search

john-hu / untitled

0 stars 0 forks source link

handle forbidden error #96

Closed john-hu closed 2 years ago

john-hu commented 2 years ago 
2021-12-16 21:18:05 [peeler.scrapy_utils.spiders.base] ERROR: <twisted.python.failure.Failure scrapy.exceptions.IgnoreRequest: Forbidden by robots.txt>
2021-12-16 21:18:05 [peeler.scrapy_utils.spiders.base] ERROR: <twisted.python.failure.Failure scrapy.exceptions.IgnoreRequest: Forbidden by robots.txt>
2021-12-16 21:18:05 [peeler.scrapy_utils.spiders.base] ERROR: <twisted.python.failure.Failure scrapy.exceptions.IgnoreRequest: Forbidden by robots.txt>
2021-12-16 21:18:05 [peeler.scrapy_utils.spiders.base] ERROR: <twisted.python.failure.Failure scrapy.exceptions.IgnoreRequest: Forbidden by robots.txt>
2021-12-10 14:18:04 [scrapy.downloadermiddlewares.robotstxt] DEBUG: Forbidden by robots.txt: <GET https://www.bettycrocker.com/register/reg-fp?returnUrl=%2frecipes%2fcourses%2fbreakfast-and-brunch-recipes%2fmuffin-recipes&isRegistrationFirstStep=false&escr=11275&RegAction=HEADERJOINLOGIN>