2020-02-16 17:55:02 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.dcard.tw/_api/posts/232906257/>
Traceback (most recent call last):
File "/Users/wyw/.local/share/virtualenvs/NewsScraping-I6zyEuYv/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
defer.returnValue((yield download_func(request=request, spider=spider)))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
沒有壞掉的網頁:
2020-02-16 18:02:39 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.coco01.today/post/1173075>
Traceback (most recent call last):
File "/Users/wyw/.local/share/virtualenvs/NewsScraping-I6zyEuYv/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
defer.returnValue((yield download_func(request=request, spider=spider)))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
dcard 的 update spider可以處理404,但如果用
ns.py update
,利用twisted一起跑所有site的時候,404 的網站似乎會直接斷在twisted connection,因此無法妥善處理。 除此之外也發現一些沒有404的網頁,也會出現以下錯誤。dcard 404 網頁:
沒有壞掉的網頁:
希望可以
ns.py update
的log 如果有遇到以上connection lost的時候,print HTTP codens.py udpate
可以處理 404