search

my8100 / scrapydweb

Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
https://github.com/my8100/files
GNU General Public License v3.0
3.17k stars 565 forks source link

Spider close status cannot show correctly #89

Closed lx1054331851 closed 5 years ago

lx1054331851 commented 5 years ago

If I click the stop button to suspend a spider, although the spider have been suspended on the server, but the stop button does not change to "start", it always show "stop", as below pictures show: image image

my8100 commented 5 years ago

Check if the job is still in the running state in http://127.0.0.1/jobs

lx1054331851 commented 5 years ago

I found some times the status can correctly show after I clicked the stop button twice. When I first clicked the stop button, log file show tips: 

scrapy.crawler] INFO: Received SIGTERM, shutting down gracefully. Send again to force

When I secondly clicked the stop button,log file show tips:

2019-08-28 03:21:54 [scrapy.core.engine] INFO: Closing spider (shutdown)
2019-08-28 03:21:54 [scrapy.extensions.logstats] INFO: Crawled 7 pages (at 7 pages/min), scraped 0 items (at 0 items/min)
2019-08-28 03:21:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www2.hm.com/zh_cn/ladies/shop-by-product/view-all/_jcr_content/main/productlisting_30ab.display.html?product-type=ladies_all&sort=newProduct&offset=7000&page-size=500> (referer: https://www2.hm.com/zh_cn/ladies/shop-by-product/view-all.html)
2019-08-28 03:21:58 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www2.hm.com/zh_cn/ladies/shop-by-product/view-all/_jcr_content/main/productlisting_30ab.display.html?product-type=ladies_all&sort=newProduct&offset=6500&page-size=500> (referer: https://www2.hm.com/zh_cn/ladies/shop-by-product/view-all.html)
2019-08-28 03:22:07 [scrapy.crawler] INFO: Received SIGTERM twice, forcing unclean shutdown
2019-08-28 03:22:07 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www2.hm.com/zh_cn/ladies/shop-by-product/view-all/_jcr_content/main/productlisting_30ab.display.html?product-type=ladies_all&sort=newProduct&offset=5500&page-size=500> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>

And last the button changes it's state from "stop" to "start".

But some time when I click the stop button twice, I check the log file, the spider has been canceled, but the button state is still "stop". So I reboot the scrapyd server, the button changes to "kill PID".

I run into the situation serveral times, today I try again, it seems return to normal!

I will try later. If i have more information, I will post here.

Thank you very much!

my8100 commented 5 years ago

Actually, the Jobs page of ScrapydWeb is dynamically generated based on the original Jobs page of Scrapyd. Always check the Scrapyd page to see whether a job is running or finished when in doubt.