Closed Akise closed 1 year ago
Pausing/resuming is a Scrapy feature, so you can do it as documented here: https://docs.scrapy.org/en/latest/topics/jobs.html#jobs-pausing-and-resuming-crawls
It would involve passing the JOBDIR setting when scheduling the crawl, e.g.
curl http://host/schedule.json -d project=myproject -d spider=somespider -d setting=JOBDIR=/path/to/dir
The cancel endpoint does the same thing as Ctrl-C from Scrapy's documentation (i.e. sending the TERM signal).
@jpmckinney When using cancel, the crawler exited directly, and did not see something similar in the log
2023-05-20 21:28:07 [scrapy.crawler]:328 INFO: Received SIGINT, shutting down gracefully. Send again to force
2023-05-20 21:28:07 [scrapy.core.engine]:406 INFO: Closing spider (shutdown)
. When running again, the jobdir is recreated to overwrite the previous state
I don't know where the problem is
This is a Scrapy issue, not a Scrapyd issue. Scrapyd has no knowledge of Scrapy's JOBDIR features. It lets Scrapy manage the JOBDIR. Scrapyd is essentially just calling scrapy crawl
with the spider settings you provide it via -d setting=MYVAR=MYVALUE
like in my example command.
I'd like to have persistence in my spider project while using scrapyd, but I don't quite know how exactly to do it.