search

crawl-blog / crawl-blog.github.io

website source of http://crawl.blog
0 stars 0 forks source link

scrapy-loop #2

Open utterances-bot opened 4 years ago

utterances-bot commented 4 years ago

Scrapy: How to loop a crawler

Explore ways to properly loop scrapy crawlers with twisted callbacks.

http://crawl.blog/scrapy-loop.html

onluha commented 4 years ago

Thanks a lot. How can I run with the code from you repo multiple spiders at once in an infinite loop?

derekang commented 4 years ago

got the same question as onluha above too! how does the deferred callback works if we have multiple spiders.