search

john-hu / untitled

0 stars 0 forks source link

Unknown lock issue #97

Open john-hu opened 2 years ago

john-hu commented 2 years ago 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select parse_state, count(*) from RECIPES_LIST group by parse_state;
0|11448
1|4712
2|1885
3|655
sqlite>
john-hu commented 2 years ago

non-200 error will causes URL locked issue.

john-hu commented 2 years ago

Lock 60, it only runs 10:

2021-12-29 06:55:02 [scrapy.core.engine] INFO: Spider opened
2021-12-29 06:55:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2021-12-29 06:55:03 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6025
2021-12-29 06:55:03 [peeler.scrapy_utils.spiders.base] INFO: locked urls count: 60
2021-12-29 06:55:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.recipetineats.com/robots.txt> (referer: None)
2021-12-29 06:55:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.recipetineats.com/pearl-barley-soup/comment-page-2/#comment-1225938> (referer: None)
2021-12-29 06:55:47 [peeler.general.spiders.general_result] INFO: site: RecipeTin Eats in en-US
2021-12-29 06:55:47 [peeler.scrapy_utils.spiders.generator_base] ERROR: URL https://www.recipetineats.com/pearl-barley-soup/comment-page-2/ does not have enough data: empty_payload
2021-12-29 06:56:03 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 0 items (at 0 items/min)
2021-12-29 06:56:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.recipetineats.com/pearl-barley-soup/comment-page-1/#comments> (referer: None)
{'downloader/request_bytes': 2542,
 'downloader/request_count': 10,
 'downloader/request_method_count/GET': 10,
 'downloader/response_bytes': 437883,
 'downloader/response_count': 10,
 'downloader/response_status_count/200': 10,
 'dupefilter/filtered': 51,
 'elapsed_time_seconds': 349.663701,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2021, 12, 28, 23, 0, 52, 669646),
 'httpcompression/response_bytes': 2248456,
 'httpcompression/response_count': 10,
 'log_count/DEBUG': 11,
 'log_count/ERROR': 9,
 'log_count/INFO': 25,
john-hu commented 2 years ago

No logs with this one:

2021-12-29 21:57:45 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.recipetineats.com/category/beef-mince-recipes/> (referer: None)
john-hu commented 2 years ago

When we received a 301, we won't print any unlock or error log. related to #105