Open Prometheus3375 opened 1 year ago
In Scrapy 2.10 RetryMiddleware was updated (PR https://github.com/scrapy/scrapy/pull/5929) to deprecate EXCEPTIONS_TO_RETRY tuple. There is an error (issue https://github.com/scrapy/scrapy/issues/6049) that makes RetryUserAgentMiddleware instances to throw AttributeError upon accessing EXCEPTIONS_TO_RETRY.
RetryMiddleware
EXCEPTIONS_TO_RETRY
RetryUserAgentMiddleware
AttributeError
To resolve this problem and add support for new RETRY_EXCEPTIONS setting, __init__ must be updated in the following way:
RETRY_EXCEPTIONS
__init__
def __init__(self, crawler: Crawler): RetryMiddleware.__init__(self, crawler.settings) RandomUserAgentBase.__init__(self, crawler) if hasattr(self, 'exceptions_to_retry'): self.EXCEPTIONS_TO_RETRY = self.exceptions_to_retry
Such solution maintains support for previous Scrapy versions.
In Scrapy 2.10
RetryMiddleware
was updated (PR https://github.com/scrapy/scrapy/pull/5929) to deprecateEXCEPTIONS_TO_RETRY
tuple. There is an error (issue https://github.com/scrapy/scrapy/issues/6049) that makesRetryUserAgentMiddleware
instances to throwAttributeError
upon accessingEXCEPTIONS_TO_RETRY
.To resolve this problem and add support for new
RETRY_EXCEPTIONS
setting,__init__
must be updated in the following way:Such solution maintains support for previous Scrapy versions.