Azure Feed exporter not working

[MathiasIconB]

I am having problems with getting the azure feed exporter plugin to work with scrapyd (https://github.com/scrapy-plugins/scrapy-feedexporter-azure-storage).

When I run this locally with scrapy it dumps the items directly to the blob storage as expected, however it does not do this when running it through a scrapyd cluster.

I am pretty sure I have the scrapy-feedexporter-azure-storage dependencies installed on the scrapyd cluster.

Here are the relevant parts of the settings.py file;

# settings.py
FEED_STORAGES = {'azure': 'scrapy_azure_exporter.AzureFeedStorage'}
FEED_EXPORT_BATCH_ITEM_COUNT = 20
AZURE_CONNECTION_STRING = 'XYZ'
FEEDS = {
    "azure://xyz.blob.core.windows.net/test/%(batch_time)s_%(batch_id)d.json": {
        "format": "json"
        }
    }

When running this code on the scrapyd cluster I get the following error in the logs, which does not appear when running it locally:

2023-10-26 12:29:31 [scrapy.extensions.feedexport] ERROR: %(batch_time)s or %(batch_id)d must be in the feed URI (file:///var/lib/scrapyd/items/myspider/myitems/49cf024e73fb11ee8586c68804063c80.jl) if FEED_EXPORT_BATCH_ITEM_COUNT setting or FEEDS.batch_item_count is specified and greater than 0. For more info see: https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-export-batch-item-count
2023-10-26 12:29:32 [scrapy.middleware] INFO: Enabled extensions:

Any help on this would be massively appreciated!

[jpmckinney]

Please check whether you are setting items_dir in your configuration file. If you are, the FEEDS is getting overwritten.

https://scrapyd.readthedocs.io/en/stable/config.html#items-dir

[MathiasIconB]

Amazing! That fixed it. Many thanks!