Pandas v2.1.0 broke Market Calendars

[davidlatte]

Greetings devs!

There is a critical bug/dependency with pandas_market_calendars and pandas itself that is causing this module to be completely broken. It appears that the new version of pandas has changed something under the hood that is no longer compatible with some of your data structures. I have verified that this is failing on both Windows and Linux machines. Please help!

Test Case

Upgrade Pandas to v2.1.0 pip install pandas --upgrade

Then simply run your own Quick Start code

import pandas_market_calendars as mcal

# Create a calendar
nyse = mcal.get_calendar('NYSE')

# Show available calendars
print(mcal.get_calendar_names())
early = nyse.schedule(start_date='2012-07-01', end_date='2012-07-10')
early

Error Log

  File "C:\Users\User\venv\Lib\site-packages\pandas_market_calendars\holidays_nyse.py", line 306, in july_5th_holiday_observance
    return datetime_index[datetime_index.year < 2013]
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'Timestamp' object is not subscriptable

[muttermachine]

I am not involved in pandas_market_calendars day to day so would not call myself one of the devs, but I am unable to reproduce in a fresh venv on Windows using python 3.11.3:

(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ python --version
Python 3.11.3

(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ pip list
Package                 Version
----------------------- -------
exchange-calendars      4.2.8
korean-lunar-calendar   0.3.1
numpy                   1.25.2
pandas                  2.1.0
pandas-market-calendars 4.2.1
pip                     22.3.1
pyluach                 2.2.0
python-dateutil         2.8.2
pytz                    2023.3
setuptools              65.5.0
six                     1.16.0
toolz                   0.12.0
tzdata                  2023.3

(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ cat example.py
import pandas_market_calendars as mcal

# Create a calendar
nyse = mcal.get_calendar('NYSE')

# Show available calendars
print(mcal.get_calendar_names())

(venv311) MINGW64 ~/code/open-source/pandas_market_calendars_env
$ python example.py
['ASX', 'BMF', 'B3', 'CFE', 'CBOE_Futures', 'CBOE_Equity_Options', 'CBOE_Index_Options', 'CME_Equity', 'CBOT_Equity', 'CME_Agriculture', 'CBOT_Agriculture', 'COMEX_Agriculture', 'NYMEX_Agriculture', 'CME_Rate', 'CBOT_Rate', 'CME_InterestRate', 'CBOT_InterestRate', 'CME_Bond', 'CBOT_Bond', 'CMEGlobex_Livestock', 'CMEGlobex_Live_Cattle', 'CMEGlobex_Feeder_Cattle', 'CMEGlobex_Lean_Hog', 'CMEGlobex_Port_Cutout', 'CMEGlobex_FX', 'CME_FX', 'CME_Currency', 'CMEGlobex_EnergyAndMetals', 'CMEGlobex_Energy', 'CMEGlobex_CrudeAndRefined', 'CMEGlobex_NYHarbor', 'CMEGlobex_HO', 'HO', 'CMEGlobex_Crude', 'CMEGlobex_CL', 'CL', 'CMEGlobex_Gas', 'CMEGlobex_RB', 'RB', 'CMEGlobex_MicroCrude', 'CMEGlobex_MCL', 'MCL', 'CMEGlobex_NatGas', 'CMEGlobex_NG', 'NG', 'CMEGlobex_Dutch_NatGas', 'CMEGlobex_TTF', 'TTF', 'CMEGlobex_LastDay_NatGas', 'CMEGlobex_NN', 'NN', 'CMEGlobex_CarbonOffset', 'CMEGlobex_CGO', 'CGO', 'C-GEO', 'CMEGlobex_NGO', 'NGO', 'CMEGlobex_GEO', 'GEO', 'CMEGlobex_Metals', 'CMEGlobex_PreciousMetals', 'CMEGlobex_Gold', 'CMEGlobex_GC', 'GC', 'CMEGlobex_SilverCMEGlobex_SI', 'SI', 'CMEGlobex_Platinum', 'CMEGlobex_PL', 'PL', 'CMEGlobex_BaseMetals', 'CMEGlobex_Copper', 'CMEGlobex_HG', 'HG', 'CMEGlobex_Aluminum', 'CMEGlobex_ALI', 'ALI', 'CMEGlobex_QC', 'QC', 'CMEGlobex_FerrousMetals', 'CMEGlobex_HRC', 'HRC', 'CMEGlobex_BUS', 'BUS', 'CMEGlobex_TIO', 'TIO', 'CME Globex Equity', 'CME Globex Fixed Income', 'CME Globex Interest Rate Products', 'EUREX', 'HKEX', 'ICE', 'ICEUS', 'NYFE', 'NYSE', 'stock', 'NASDAQ', 'BATS', 'DJIA', 'DOW', 'IEX', 'Investors_Exchange', 'JPX', 'LSE', 'OSE', 'SIFMAUS', 'SIFMA_US', 'Capital_Markets_US', 'Financial_Markets_US', 'Bond_Markets_US', 'SIFMAUK', 'SIFMA_UK', 'Capital_Markets_UK', 'Financial_Markets_UK', 'Bond_Markets_UK', 'SIFMAJP', 'SIFMA_JP', 'Capital_Markets_JP', 'Financial_Markets_JP', 'Bond_Markets_JP', 'SIX', 'SSE', 'TSX', 'TSXV', 'BSE', 'NSE', 'TASE', 'AIXK', 'ASEX', 'BVMF', 'CMES', 'IEPA', 'XAMS', 'XASX', 'XBKK', 'XBOG', 'XBOM', 'XBRU', 'XBSE', 'XBUD', 'XBUE', 'XCBF', 'XCSE', 'XDUB', 'XFRA', 'XETR', 'XHEL', 'XHKG', 'XICE', 'XIDX', 'XIST', 'XJSE', 'XKAR', 'XKLS', 'XKRX', 'XLIM', 'XLIS', 'XLON', 'XMAD', 'XMEX', 'XMIL', 'XMOS', 'XNYS', 'XNZE', 'XOSL', 'XPAR', 'XPHS', 'XPRA', 'XSAU', 'XSES', 'XSGO', 'XSHG', 'XSTO', 'XSWX', 'XTAE', 'XTAI', 'XTKS', 'XTSE', 'XWAR', 'XWBO', 'us_futures', '24/7', '24/5']

Is there anything else in your environment?

[davidlatte]

Can you go one step farther in your test?

early = nyse.schedule(start_date='2012-07-01', end_date='2012-07-10')
early

This is the step that is throwing the error. Sorry if that wasn't clear.

Here is my own Windows environment for reference.

(venv) PS C:\Users\User\code\> python --version
Python 3.11.4
(venv) PS C:\Users\User\code\> pip list
Package                 Version
----------------------- -------
exchange-calendars      4.2.8
korean-lunar-calendar   0.3.1
numpy                   1.25.2
pandas                  2.1.0
pandas-market-calendars 4.2.1
pip                     23.1.2
pyluach                 2.2.0
python-dateutil         2.8.2
pytz                    2023.3
setuptools              65.5.0
six                     1.16.0
toolz                   0.12.0
tzdata                  2023.3

[maread99]

This looks the same as this issue with exchange_calendars.

[cfaaron]

My apologies, I get the error when I go a step further. There appears to be a change in how pandas handles Index.map() it appears to no longer pass through the index and then fallback to value-by-value calls. The below code run in pandas 2.0.3 but not in 2.1.0 (where the assertion fails):

import pandas as pd
from pandas.tseries.holiday import Holiday

def _observance(d):
    assert isinstance(d, pd.DatetimeIndex)
    return d[d.year < 2013]

h = Holiday(
    # When July 4th is a Thursday, the next day is a half day prior to 2013.
    # Since 2013 the early close is on Wednesday and Friday is a full day
    "Fridays after Independence Day prior to 2013",
    month=7,
    day=5,
    days_of_week=(4,),  # FRIDAY
    observance=_observance,
    start_date=pd.Timestamp("1996-01-01"),
)

result = h.dates(pd.Timestamp("1991-01-01"), pd.Timestamp("2023-01-01"))
print(result)

I believe there are only two places where the observance expects an index (both july_5th_holiday_observance defs). However, there are a number of other unit tests failing with the latest pandas due to other small changes. I have most of them passing, but am trusting the tests to guide me.

[rsheftel]

Fixed in PR #282