`Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True` when `ignore_tz=False`

mreiche commented 9 months ago

Describe bug

I get the error Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True when downloading history data in PROD.

My PROD setup:

Linux AMD 64
Timezone: CET
Python 3.11.5
yfinance 0.2.35
pandas 2.1.4

Simple code that reproduces your problem

~~I cannot reproduce the error in my DEV setup, but~~ the pseudo code leading to this error, is the following:

yf.download(
    tickers=" ".join(symbols),
    period=period.value,
    interval=interval.value,
    group_by="tickers",
    prepost=True,
    ignore_tz=False
)

My DEV setup:

Mac ARM 64
Timezone: CET
Python: 3.11.6
yfinance 0.2.35
pandas 2.1.4

When setting ignore_tz=True the error disappears (leading to another error that's not covered by this issue here).

Debug log

  File "/home/backend/yahoo/utils/__init__.py", line 41, in __init__
    self.__data = yf.download(
                  ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/utils.py", line 103, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/yfinance/multi.py", line 225, in download
    data.index = _pd.to_datetime(data.index)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/tools/datetimes.py", line 1121, in to_datetime
    result = convert_listlike(arg, format, name=arg.name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/tools/datetimes.py", line 490, in _convert_listlike_datetimes
    result, tz_parsed = objects_to_datetime64ns(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py", line 2346, in objects_to_datetime64ns
    result, tz_parsed = tslib.array_to_datetime(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "tslib.pyx", line 403, in pandas._libs.tslib.array_to_datetime
  File "tslib.pyx", line 552, in pandas._libs.tslib.array_to_datetime
  File "tslib.pyx", line 480, in pandas._libs.tslib.array_to_datetime
  File "conversion.pyx", line 716, in pandas._libs.tslibs.conversion.convert_timezone
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True, at position 4

Bad data proof

No response

`yfinance` version

0.2.35 Previous tested version 0.2.32 works fine.

Python version

3.11.5

Operating system

Linux 6c9a93488545 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 GNU/Linux Podman 3.4.4

mreiche commented 9 months ago

I was now able to reproduce this in my DEV setup. It seems to be related to specific symbols. I'm investigating. The change came with 4a5616d5c40da3dcd1350282329869fe19aea6e9

ValueRaider commented 9 months ago

Just a thought: does this error occur when tickers from different timezones? Mixing timezones can cause problems in Pandas. Easily solved though.

ValueRaider commented 9 months ago

Another thought: does downgrading Pandas to below 2.1 help? That introduced a subtle datetime bug that can propagate and cause chaos.

mreiche commented 2 months ago

I'm experiencing this bug again in Pandas. In multi.py line 214

data.index = _pd.to_datetime(data.index)

where data.index is:

Index([2024-08-07 09:05:00+02:00, 2024-08-07 10:40:00+02:00,
       2024-08-07 13:15:00+02:00, 2024-08-07 13:20:00+02:00,
       2024-08-07 09:30:00-04:00, 2024-08-07 09:35:00-04:00,
       2024-08-07 09:40:00-04:00, 2024-08-07 09:45:00-04:00,
       2024-08-07 09:50:00-04:00, 2024-08-07 09:55:00-04:00,
       2024-08-07 10:00:00-04:00, 2024-08-07 10:05:00-04:00,
       2024-08-07 10:10:00-04:00, 2024-08-07 10:15:00-04:00,
       2024-08-07 10:20:00-04:00, 2024-08-07 10:25:00-04:00,
       2024-08-07 10:30:00-04:00, 2024-08-07 10:35:00-04:00,
       2024-08-07 10:40:00-04:00, 2024-08-07 10:45:00-04:00,
       2024-08-07 10:50:00-04:00, 2024-08-07 10:55:00-04:00,
       2024-08-07 11:00:00-04:00, 2024-08-07 11:05:00-04:00,
       2024-08-07 11:10:00-04:00, 2024-08-07 11:15:00-04:00,
       2024-08-07 11:20:00-04:00, 2024-08-07 11:25:00-04:00,
       2024-08-07 11:30:00-04:00, 2024-08-07 11:35:00-04:00,
       2024-08-07 11:40:00-04:00, 2024-08-07 11:45:00-04:00,
       2024-08-07 11:50:00-04:00, 2024-08-07 11:55:00-04:00,
       2024-08-07 12:00:00-04:00, 2024-08-07 12:05:00-04:00,
       2024-08-07 12:10:00-04:00, 2024-08-07 12:15:00-04:00,
       2024-08-07 12:20:00-04:00, 2024-08-07 12:25:00-04:00,
       2024-08-07 12:30:00-04:00, 2024-08-07 12:35:00-04:00,
       2024-08-07 12:40:00-04:00, 2024-08-07 12:45:00-04:00,
       2024-08-07 12:50:00-04:00],
      dtype='object')

It fails with: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True, at position 4

Position 4 has a timezone of -4.

ranaroussi / yfinance