I hardcoded Montreal as using a proxy. Since the need for a proxy depends on the server, we can instead implement the list of spiders needing a proxy as an environment variable, and configure the custom settings on the base spider.
Would need to look into how/when Scrapy reads the custom_settings class attribute, as we’ll want subclasses to be able to add to it.
I hardcoded Montreal as using a proxy. Since the need for a proxy depends on the server, we can instead implement the list of spiders needing a proxy as an environment variable, and configure the custom settings on the base spider.
Would need to look into how/when Scrapy reads the
custom_settingsclass attribute, as we’ll want subclasses to be able to add to it.