[BUG]: parquet containing ConvertedType TIMESTAMP_MILLIS may throw IndexOutOfRangeException while reading.

[artnim]

Library Version

4.24.0

OS

Ubuntu Linux

OS Architecture

64 bit

How to reproduce?

1. Create a parquet with timeseries as datetime64[ms]

import pandas as pd

pd.set_option('io.parquet.engine', 'fastparquet')

start_time = pd.Timestamp('2024-08-21T00:00:00+02:00')
end_time = pd.Timestamp('2024-08-22T00:00:00+02:00')
series = pd.date_range(start_time, end_time, freq=pd.Timedelta(minutes=15))[:-1]

series = series.tz_convert('UTC').tz_localize(None).astype("datetime64[ms]")

df = pd.DataFrame({'datetime': series})

print(df)
print(df.dtypes)

df.to_parquet('./test.parquet')

2. Open the resulting test.parquet using floor

The same works like a charm reading a parquet with created with the library. So I suggest a dependency with the fastparquet engine used my site.

Failing test

No response

[artnim]

Meanwhile, I can confirm that the problem is caused by what fastparquet writes. The problem does not exist with pyarrow.