scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
524 stars 94 forks source link

spidermon.utils.zyte.client uses only project setting module for settings. #410

Closed HassanQamar07 closed 11 months ago

HassanQamar07 commented 11 months ago

We came across an issue where we were configuring ZyteJobsComparisonMonitor and were getting authentication errors. We were using Zyte SC project settings to set the right keys but still, the authentication issue was not resolved. I dug into the code and checked how the Client is fetching the setting and found it is using get_project_settings from scrapy.utils.project at https://github.com/scrapinghub/spidermon/blob/e6509dd0523acded0d6cf7dd3422a9581a99815a/spidermon/utils/zyte.py#L43, this method only reads setting set in project setting module due to which the settings set at SC project settings page were not being used.

Due to this behaviour, the Zyte Client does not have access to custom settings, SC project/spider settings or cmd line settings.