Closed fnesveda closed 1 year ago
This adds a Python + Scrapy template.
It uses the default Scrapy project structure, and is based on an example straight from Scrapy: https://github.com/scrapy/quotesbot
It's runnable both as an actor, and also directly via Scrapy: scrapy crawl quotes -a tag=humor -o quotes.json
scrapy crawl quotes -a tag=humor -o quotes.json
The Scrapy project can actually remain completely untouched, we can just add a few files around it to make it into an actor.
Once we agree on the code I'd probably publish it as an example actor too.
BTW, figuring out the logging setup was a pain in the ass.
Also, the Scrapy example is published with the MIT license, we should include it somewhere here in the repo, to be completely correct.
This adds a Python + Scrapy template.
It uses the default Scrapy project structure, and is based on an example straight from Scrapy: https://github.com/scrapy/quotesbot
It's runnable both as an actor, and also directly via Scrapy:
scrapy crawl quotes -a tag=humor -o quotes.jsonThe Scrapy project can actually remain completely untouched, we can just add a few files around it to make it into an actor.
Once we agree on the code I'd probably publish it as an example actor too.
BTW, figuring out the logging setup was a pain in the ass.
Also, the Scrapy example is published with the MIT license, we should include it somewhere here in the repo, to be completely correct.