alecxe / scrapy-fake-useragent

Random User-Agent middleware based on fake-useragent
MIT License
686 stars 98 forks source link

Add support for Scrapy 2.10 and `RETRY_EXCEPTIONS` setting #41

Open Prometheus3375 opened 1 year ago

Prometheus3375 commented 1 year ago

In Scrapy 2.10 RetryMiddleware was updated (PR https://github.com/scrapy/scrapy/pull/5929) to deprecate EXCEPTIONS_TO_RETRY tuple. There is an error (issue https://github.com/scrapy/scrapy/issues/6049) that makes RetryUserAgentMiddleware instances to throw AttributeError upon accessing EXCEPTIONS_TO_RETRY.

To resolve this problem and add support for new RETRY_EXCEPTIONS setting, __init__ must be updated in the following way:

    def __init__(self, crawler: Crawler):
        RetryMiddleware.__init__(self, crawler.settings)
        RandomUserAgentBase.__init__(self, crawler)

        if hasattr(self, 'exceptions_to_retry'):
            self.EXCEPTIONS_TO_RETRY = self.exceptions_to_retry

Such solution maintains support for previous Scrapy versions.