search

john-hu / untitled

0 stars 0 forks source link

handle dupfilter error #106

Open john-hu opened 2 years ago

john-hu commented 2 years ago 
{'downloader/request_bytes': 2856,
 'downloader/request_count': 11,
 'downloader/request_method_count/GET': 11,
 'downloader/response_bytes': 415708,
 'downloader/response_count': 11,
 'downloader/response_status_count/200': 10,
 'downloader/response_status_count/404': 1,
 'dupefilter/filtered': 50,
2021-12-29 20:55:42 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.recipetineats.com/slow-cooked-shredded-beef-ragu-pasta/comment-page-30/#comment-1245458> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)