TypeError: Retry.__init__() got an unexpected keyword argument 'method_whitelist' (urllib3 upgrade)

Bentroen commented 11 months ago

After upgrading the dependencies in my project, I've started facing the following exception:

Traceback (most recent call last):
<omitted>
  File "...\application\trends.py", line 315, in get_related_queries
    self._build_payload(terms)
  File "...\application\trends.py", line 214, in _build_payload    
    self.trend_req.build_payload(
  File "...\.venv\lib\site-packages\pytrends\request.py", line 189, in build_payload
    self._tokens()
  File "...\.venv\lib\site-packages\pytrends\request.py", line 195, in _tokens
    widget_dicts = self._get_data(
  File "...\.venv\lib\site-packages\pytrends\request.py", line 124, in _get_data
    retry = Retry(total=self.retries, read=self.retries,
TypeError: Retry.__init__() got an unexpected keyword argument 'method_whitelist'

Turns out the error started after urllib3 was upgraded from 1.26.12 to 2.0.4, as a result of upgrading Requests to anything >2.30.0.

The fix for projects using Pytrends is to downgrade the dependency manually, i.e. pip install urllib3<2 or poetry add urllib3<2.

Investigation

From the urllib3 2.0.0 changelog, here's the cause of the issue:

Removed deprecated Retry options method_whitelist, DEFAULT_REDIRECT_HEADERS_BLACKLIST (https://github.com/urllib3/urllib3/issues/2086).

A workaround would be to specify a maximum version for urllib3 in requirements.txt, such as urllib3<2 (this is also the approach recommended by the Requests library in the version below). Though, since it is not a direct dependency of pytrends, but of the Requests package, it might be more complicated.

Support for urllib3 v2.0 was added in Requests v2.30.0 (changelog), so pinning its requirement as requests<2.30 could work as well (though there's no reason for pinning it as further versions are still compatible with urllib 1.x, and pinning urllib3 is also the approach recommended by Requests in the linked page).

In the long run, it might be easier/worth it to upgrade the module to be compatible with urllib 2.x. A full migration guide is available here, though the line mentioned in this issue could likely be the only necessary change.

mikedbjones commented 11 months ago

The fix for projects using Pytrends is to downgrade the dependency manually, i.e. pip install urllib3<2

Thank you for this, it helped me with the same error. On Ubuntu add quotes: pip install 'urllib3<2'

Helldez commented 11 months ago

Does this also solve the issues of the previous threads about the 500 and 429 errors for timeframes less than or equal to a week?

GeneralMills / pytrends