BUG: pandas.errors.LossySetitemError when backtesting Freqtrade

JPeetz commented 4 months ago

Pandas version checks

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

2024-05-29 11:45:19,821 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1855, in shift
    casted = np_can_hold_element(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/dtypes/cast.py", line 1936, in np_can_hold_element
    raise LossySetitemError
pandas.errors.LossySetitemError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 42, in main
    return_code = args['func'](args)
                  ^^^^^^^^^^^^^^^^^^
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 58, in start_backtesting
    backtesting.start()
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 1410, in start
    min_date, max_date = self.backtest_one_strategy(strat, data, timerange)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 1344, in backtest_one_strategy
    results = self.backtest(
              ^^^^^^^^^^^^^^
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 1222, in backtest
    data: Dict = self._get_ohlcv_as_lists(processed)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 401, in _get_ohlcv_as_lists
    [nan], [0 if not tag_col else None]).shift(1)
                                         ^^^^^^^^
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/generic.py", line 11225, in shift
    new_data = self._mgr.shift(periods=periods, fill_value=fill_value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/internals/base.py", line 312, in shift
    return self.apply_with_block("shift", periods=periods, fill_value=fill_value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 363, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1859, in shift
    nb = self.coerce_to_target_dtype(fill_value)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ftuser/.local/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 490, in coerce_to_target_dtype
    raise AssertionError(
AssertionError: Something has gone wrong, please report a bug at https://github.com/pandas-dev/pandas/issues

Issue Description

The issue occurs when running a Backtest with the Freqtrade Trading Bot, attempting to backtest a short futures strategy.

Resources: https://github.com/ssssi/freqtrade_strs/blob/82baabd2099b5f592458678554ca7162148f1d7a/binance/BinHV27/short/BinHV27_short.py

https://github.com/ssssi/freqtrade_strs/blob/82baabd2099b5f592458678554ca7162148f1d7a/binance/BinHV27/short/BinHV27_short.json

https://github.com/freqtrade/freqtrade

Expected Behavior

To return backtesting result, as the bot does for spot trading backtests.

Installed Versions

sudo -H pip3 install pandas [sudo] password for jpeetz: Collecting pandas Downloading pandas-2.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB) |████████████████████████████████| 12.4 MB 1.1 MB/s Collecting pytz>=2020.1 Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB) |████████████████████████████████| 505 kB 33.5 MB/s Collecting python-dateutil>=2.8.2 Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) |████████████████████████████████| 229 kB 22.9 MB/s Collecting numpy>=1.20.3; python_version < "3.10" Downloading numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB) |████████████████████████████████| 17.3 MB 36.2 MB/s Collecting tzdata>=2022.1 Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB) |████████████████████████████████| 345 kB 30.6 MB/s Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas) (1.14.0) Installing collected packages: pytz, python-dateutil, numpy, tzdata, pandas Successfully installed numpy-1.24.4 pandas-2.0.3 python-dateutil-2.9.0.post0 pytz-2024.1 tzdata-2024.1

Aloqeely commented 4 months ago

Thanks for the report! Could you provide a minimal bug report which reproduces this error without the use of the freqtrade library?

mroeschke commented 1 month ago

Closing as a reproducible example is needed

pandas-dev / pandas