ReadTimeoutError on fc.yahoo.com

[lucas03]

Describe bug

I can not reproduce on my local macbook, but on my hetzner server I see a lot of timeouts to fc.yahoo.com. I found an issue that says fc.yahoo.com got to some blacklist?

Would it be possible to use different domain for getting the cookie. Possibly make retries and timeout configurable?

Simple code that reproduces your problem

yfinance.Ticker('IWTNF').isin

Debug log

{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963145.5295649}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963151.180321}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963157.4302988}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963164.8873549}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963174.7397318}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963186.1650689}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963191.8163652}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963198.0603182}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963205.5079286}
{"event": "stdlib_log", "symbol": "IWTNF", "logger": "urllib3.connectionpool", "message": "Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError(\"HTTPSConnectionPool(host='fc.yahoo.com', port=443): Read timed out. (read timeout=5)\")': /", "level": "warning", "timestamp": 1701963215.3532467}
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/app/app/stocks/updater.py", line 318, in update_stock_from_yahoo
    if not stock.isin and ticker.isin and ticker.isin != "-":
                          ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/ticker.py", line 106, in isin
    return self.get_isin()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/base.py", line 2024, in get_isin
    if self._quote.info is None:
       ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/scrapers/quote.py", line 572, in info
    self._fetch(self.proxy)
  File "/usr/local/lib/python3.11/site-packages/yfinance/scrapers/quote.py", line 602, in _fetch
    result = self._data.get_raw_json(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/data.py", line 399, in get_raw_json
    response = self.get(url, user_agent_headers=user_agent_headers, params=params, proxy=proxy, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/utils.py", line 108, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/data.py", line 378, in get
    request_args['cookies'] = {cookie.name: cookie.value}
                               ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'name'

Bad data proof

No response

yfinance version

0.2.33

Python version

3.11.6

Operating system

python:3.11.6-slim-bookworm

[ValueRaider]

Already has a second strategy for if first fails but never used if YF blows up. See if you can add handling for that error.

[lucas03]

Hmm, I don't really understand that code part and not sure how that urllib ReadTimeoutError is mapped to requests error?

When I tried to read that strategy code, it already fails on second try (under if response.status_code >= 400). 😕

[RudyNL]

There are also issues with yahooquery. See fc.yahoo.com not responding. You might try using finance.yahoo.com instead of fc.yahoo.com.

[ValueRaider]

@lucas03 Copy the if-statement from line #353 to #377. Should fix exception but I expect data fetch will fail because cookie fetch failed.

The cycling is messy because needs to handle concurrent threads. But hopefully clear enough to add a third strategy.

[roscherr]

I was able to get this working with the basic cookie strategy by disabling retrieval of cached cookies, changing the domain from fc.yahoo.com to guce.yahoo.com/consent, and then handling the redirect by choosing the original response in the response.history. Make these changes in yfinance/data.py: L146 -> if cookie_dict['age'] > datetime.timedelta(days=0): L163: -> url='https://guce.yahoo.com/consent', L168 -> if len(response.history) > 0: response = response.history[0]

[ValueRaider]

@roscherr #1084

[mmojz642]

@roscherr did first two steps in version '0.2.36' and this solved the issue for me.