Open andreawwenyi opened 4 years ago
This issue is related to #70.
ns.py
( which use CrawlerRunner
instead of CrawlerProcess
) behaves the similar way. It was previously thought that CrawlerRunner
threw twisted.web._newclient.ResponseNeverReceived
when the dcard posts is 404
. However, after much testing & reading the log, I've found that many of the dcard posts that result in such error are actually active.
It also happens a lot on the following sites on discover recently:
Error downloading <GET http://www.itaiwannews.cn/>
Error downloading <GET http://www.tailian.org.cn/>
Error downloading <GET http://www.whb.cn/>
Since it seems to only happen on a subset of the sites, I would guess it is more of some problems on the sites' ends. I set delay = 2
for these sites for now. Let's see if things improve. Also I'm not sure why RetryMiddleware is not working here. Or is it?
python3 execute_spider.py --update
has repeatedly shown errors like this one when loading active urls. When running update on only 1 site (in the following example, ETtoday, would bepython3 site.py update 102
), we'd not see this error and the article can be updated as expected. The problem therefore is believed to be related toCrawlerProcess
used inexecute_spider.py
.Reference: