search

scrapinghub / scrapyrt

HTTP API for Scrapy spiders
BSD 3-Clause "New" or "Revised" License
832 stars 162 forks source link

can scrapyrt be used to schedule scrapes? #146

Closed doverradio closed 7 months ago

doverradio commented 1 year ago

I am using scrapyrt extensively by just sending requests from another server where that other server has been using crontab.

However, I began wondering if I could just set scrapyrt to a schedule instead of having to fire all these requests.

This github repo states "It starts HTTP server allowing you to schedule spiders and get spider output in JSON."

However, that's the only mention of "schedule spiders".

Is there a way?

ankitjavalkar commented 1 year ago

scrapyd would better suit your use case. See https://scrapyd.readthedocs.io/en/latest/overview.html#scheduling-a-spider-run