pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.83k stars 18k forks source link

BUG: Cannot `shift` Intervals that are not `closed='right'` (the default) #60389

Open WillMorrisonEnspi opened 20 hours ago

WillMorrisonEnspi commented 20 hours ago

Pandas version checks

Reproducible Example

import pandas as pd
pd.Series([pd.Interval(1, 2, closed='left'), pd.Interval(2, 3, closed='left')]).shift(1)

Issue Description

This line in shift creates an empty IntervalArray without specifying which side the intervals are closed on. When that array and the one being shifted get concatenated, the following exception is raised:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wmorrison/.local/share/pyenv/versions/3.11.5/envs/3.11.5-enspired@aws_lambda/lib/python3.11/site-packages/pandas/core/generic.py", line 11228, in shift
    new_data = self._mgr.shift(periods=periods, fill_value=fill_value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wmorrison/.local/share/pyenv/versions/3.11.5/envs/3.11.5-enspired@aws_lambda/lib/python3.11/site-packages/pandas/core/internals/base.py", line 312, in shift
    return self.apply_with_block("shift", periods=periods, fill_value=fill_value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wmorrison/.local/share/pyenv/versions/3.11.5/envs/3.11.5-enspired@aws_lambda/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 363, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wmorrison/.local/share/pyenv/versions/3.11.5/envs/3.11.5-enspired@aws_lambda/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 2020, in shift
    new_values = self.values.T.shift(periods=periods, fill_value=fill_value).T
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wmorrison/.local/share/pyenv/versions/3.11.5/envs/3.11.5-enspired@aws_lambda/lib/python3.11/site-packages/pandas/core/arrays/interval.py", line 1097, in shift
    return self._concat_same_type([a, b])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wmorrison/.local/share/pyenv/versions/3.11.5/envs/3.11.5-enspired@aws_lambda/lib/python3.11/site-packages/pandas/core/arrays/interval.py", line 1045, in _concat_same_type
    raise ValueError("Intervals must all be closed on the same side.")
ValueError: Intervals must all be closed on the same side.

Expected Behavior

The following pd.Series[Interval] should be returned, closed on the same side as the original Series

0           NaN
1    [1.0, 2.0)
dtype: interval

Installed Versions

INSTALLED VERSIONS ------------------ commit : 0691c5cf90477d3503834d983f69350f250a6ff7 python : 3.11.5 python-bits : 64 OS : Linux OS-release : 6.8.0-48-generic Version : #48~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 7 11:24:13 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.2.3 numpy : 2.1.3 pytz : 2024.2 dateutil : 2.9.0.post0 pip : 24.3.1 Cython : None sphinx : None IPython : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 blosc : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : None html5lib : None hypothesis : None gcsfs : None jinja2 : 3.1.4 lxml.etree : None matplotlib : 3.9.2 numba : None numexpr : None odfpy : None openpyxl : 3.1.5 pandas_gbq : None psycopg2 : None pymysql : None pyarrow : None pyreadstat : None pytest : 8.3.3 python-calamine : None pyxlsb : None s3fs : None scipy : 1.14.1 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlsxwriter : None zstandard : None tzdata : 2024.2 qtpy : None pyqt5 : None
rhshadrach commented 12 hours ago

Thanks for the report! Further investigations and PRs to fix are welcome.

veronicabenedict commented 8 hours ago

take