Add support for `RETRY_EXCEPTIONS` setting

alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent

MIT License

686 stars 98 forks source link

Add support for `RETRY_EXCEPTIONS` setting #42

Open Prometheus3375 opened 1 year ago

Prometheus3375 commented 1 year ago

Resolves #41.

I tested locally with Scrapy 2.9.0, 2.10.0 and 2.11.0. 2.10 is the version with exception upon accessing EXCEPTIONS_TO_RETRY. In 2.11 this error was fixed.

On all versions middleware does not fail when process_exception is called. On 2.10+ the middleware also respects setting RETRY_EXCEPTIONS.

I am not sure how to add tests with specific scrapy version.

Prometheus3375 commented 12 months ago

@alecxe hello. Can you review this one and release a new version to PyPI?