Closed MathiasIconB closed 9 months ago
Please check whether you are setting items_dir
in your configuration file. If you are, the FEEDS
is getting overwritten.
https://scrapyd.readthedocs.io/en/stable/config.html#items-dir
Amazing! That fixed it. Many thanks!
I am having problems with getting the azure feed exporter plugin to work with scrapyd (https://github.com/scrapy-plugins/scrapy-feedexporter-azure-storage).
When I run this locally with scrapy it dumps the items directly to the blob storage as expected, however it does not do this when running it through a scrapyd cluster.
I am pretty sure I have the scrapy-feedexporter-azure-storage dependencies installed on the scrapyd cluster.
Here are the relevant parts of the settings.py file;
When running this code on the scrapyd cluster I get the following error in the logs, which does not appear when running it locally:
Any help on this would be massively appreciated!