QST: FutureWarning: Resampling with a PeriodIndex is deprecated, how to resample now?

andreas-wolf commented 10 months ago

Research

[X] I have searched the [pandas] tag on StackOverflow for similar questions.
[X] I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/77862775/pandas-2-2-futurewarning-resampling-with-a-periodindex-is-deprecated

Question about pandas

Pandas version 2.2 raises a warning when using this code:

import pandas as pd

df = pd.DataFrame.from_dict({"something": {pd.Period("2022", "Y-DEC"): 2.5}})
# FutureWarning: Resampling with a PeriodIndex is deprecated.
# Cast index to DatetimeIndex before resampling instead.
print(df.resample("M").ffill())

#          something
# 2022-01        2.5
# 2022-02        2.5
# 2022-03        2.5
# 2022-04        2.5
# 2022-05        2.5
# 2022-06        2.5
# 2022-07        2.5
# 2022-08        2.5
# 2022-09        2.5
# 2022-10        2.5
# 2022-11        2.5
# 2022-12        2.5

This does not work:

df.index = df.index.to_timestamp()
print(df.resample("M").ffill())

#             something
# 2022-01-31        2.5

I have PeriodIndex all over the place and I need to resample them a lot, filling gaps with ffill. How to do this with Pandas 2.2?

MarcoGorelli commented 10 months ago

This comes from #55968 , and here's the relevant issue https://github.com/pandas-dev/pandas/issues/53481

I'd suggest to create a datetimeindex and ffill

matteo-zanoni commented 9 months ago

I just hit this too. For upsampling the PeriodIndex had a different result than the DatetimeIndex, in particular PeriodIndex would create a row for each new period in the old period:

import pandas as pd

s = pd.Series(1, index=pd.period_range(pd.Timestamp(2024, 1, 1), pd.Timestamp(2024, 1, 2), freq="d"))
s.resample("1h").ffill()

This would create a series including ALL hours of 2024-01-02.

If, instead, we first convert to DatetimeIndex:

import pandas as pd

s = pd.Series(1, index=pd.period_range(pd.Timestamp(2024, 1, 1), pd.Timestamp(2024, 1, 2), freq="d"))
d.index = d.index.to_timestamp()
s.resample("1h").ffill()

The output will only contain 1 hour of 2024-01-02. Even if the series is constructed directly with a DatetimeIndex (even one containing the frequency information) the result is the same:

import pandas as pd

s = pd.Series(1, index=pd.date_range(pd.Timestamp(2024, 1, 1), pd.Timestamp(2024, 1, 2), freq="d"))
s.resample("1h").ffill()

Will there be no way to obtain the old behaviour of PeriodIndex in future versions? IMO upsampling is quite common and the way PeriodIndex implemented it is more usefull. It would be a shame to loose it.

andreas-wolf commented 8 months ago

IMO upsampling is quite common and the way PeriodIndex implemented it is more usefull. It would be a shame to loose it.

I agree. Now you'll need to do reindexing manually, while with periodIndex this was a one-liner. Furthermore resampling with a datetime index seems to change the data type (a bug?). Here some sample code:

import pandas as pd

# some sample data
data = {2023: 1, 2024: 2}
df = pd.DataFrame(list(data.values()), index=pd.PeriodIndex(data.keys(), freq="Y"))

# Old style resampling, just a one-liner
old_style_resampling = df.resample("M").ffill()
print(old_style_resampling)
print(type(old_style_resampling.iloc[0][0]))

# Convert index to DatetimeIndex
df.index = pd.to_datetime(df.index.start_time)
last_date = df.index[-1] + pd.offsets.YearEnd()
df_extended = df.reindex(
    df.index.union(pd.date_range(start=df.index[-1], end=last_date, freq="D"))
).ffill()
new_style_resampling = df_extended.resample("ME").ffill()
print(new_style_resampling)
print(type(new_style_resampling.iloc[0][0]))

I also opt for keeping the periodIndex resampling.

ChadFulton commented 8 months ago

Related to my comments in #56588, I think that this is another example where Period is being deprecated too fast without a clear replacement in mind.

jbrockmendel commented 8 months ago

Are all the relevant cases about upsampling and never downsampling? A big part of the motivation for deprecating was that PeriodIndexResampler._downsample is deeply broken in a way that didn't seem worth fixing. Potentially we could just deprecate downsampling and not upsampling?

andreas-wolf commented 7 months ago

The upsampling example is just the one where it's very obvious what will be missing when periodindex resampling won't work any more.

When downsampling would not work anymore I would have to convert the index, downsample and convert the index back again. Does not sound very compelling.

The period index resampling (up and down) is very convenient when one has to combine different data sources in days, months, quarters and years. I can't remember a project where I did not use period resampling. The convenience was always an argument to use pandas instead of other libraries like polars where one has to handle all the conversions yourself.

From my point of view the PeriodIndex was always one of the great things about Pandas.

I have very limited experience with Pandas internals, so I don't understand how downsampling can be deeply broken so that it's not worth fixing when "just" converting to a datetime index would fix it? Can't the datetime indexing be used internally to fix it?

MarcosYanase commented 7 months ago

I agree with @andreas-wolf. My projects have a lot of dataframes using PeriodIndex, with differents frequencies and resampling is very very useful tool for calculation. Keeping it at least for upsampling would be excellent, but for downsampling a workaround (like Andreas posted) is not trivial. What are the bugs related to PeriodIndexResampler._downsample?

MarcosYanase commented 2 months ago

Any news about this issue?

The deprecation was due this #53481, correct @jbrockmendel ?

Some of this is because in PeriodIndexResampler._downsample (which many methods go through) we just return self.asfreq() in many cases which is very much wrong.

https://github.com/pandas-dev/pandas/blob/c375533d670a7114c36ebb114c01ec7d57b92753/pandas/core/resample.py#L1800C1-L1813C33

        if is_subperiod(ax.freq, self.freq):
            # Downsampling
            return self._groupby_and_aggregate(how, **kwargs)
        elif is_superperiod(ax.freq, self.freq):
            if how == "ohlc":
                # GH #13083
                # upsampling to subperiods is handled as an asfreq, which works
                # for pure aggregating/reducing methods
                # OHLC reduces along the time dimension, but creates multiple
                # values for each period -> handle by _groupby_and_aggregate()
                return self._groupby_and_aggregate(how)
            return self.asfreq()
        elif ax.freq == self.freq:
            return self.asfreq()

I'm naively changing the lines "return self.asfreq()" to "return super()._downsample(how, **kwargs)", delivering the responsability to DatetimeIndexResampler._downsample. It works for the example in #53481 and other tests that I'm doing (using other downsampling methods), giving similar outputs to DatetimeIndexResampler._downsample. But I'm sure it's not so simple. Could you please give more examples where PeriodIndexResampler._downsample continues to be broken?

jbrockmendel commented 2 months ago

I don't have the bandwidth to give you a thorough answer. What I can tell you is that there are no plans to enforce this deprecation in 3.0.

MarcosYanase commented 2 months ago

Is there an option to not deprecate resample with PeriodIndex? If yes, how is the process? We fix the issues with it first and delete the FutureWarning, or do both simultaneously?

I think it's possible to fix the example in #53481 doing what I wrote above:

if is_subperiod(ax.freq, self.freq):
    # Downsampling
    return self._groupby_and_aggregate(how, **kwargs)
elif is_superperiod(ax.freq, self.freq):
    if how == "ohlc":
        # GH #13083
        # upsampling to subperiods is handled as an asfreq, which works
        # for pure aggregating/reducing methods
        # OHLC reduces along the time dimension, but creates multiple
        # values for each period -> handle by _groupby_and_aggregate()
        return self._groupby_and_aggregate(how)
    return super()._downsample(how, **kwargs) #fixed here, it was return self.asfreq()
elif ax.freq == self.freq:
    return super()._downsample(how, **kwargs) #fixed here, it was return self.asfreq()
raise IncompatibleFrequency(
    f"Frequency {ax.freq} cannot be resampled to {self.freq}, "
    "as they are not sub or super periods"
)

About https://github.com/pandas-dev/pandas/pull/58021#issuecomment-2027782313:

test_resample_nat_index_series (reason="Don't know why this fails"): https://github.com/pandas-dev/pandas/blob/e4956ab403846387a435cd7b3a8f36828c23c0c7/pandas/tests/resample/test_base.py#L229) -- First error: ValueError("for Period, please use 'M' instead of 'ME'") -- Doing it, it will fail but we could fix it modifying https://github.com/pandas-dev/pandas/blob/e4956ab403846387a435cd7b3a8f36828c23c0c7/pandas/core/resample.py#L1643 to
```
        if isinstance(obj.index, PeriodIndex):
            obj.index = PeriodIndex(obj.index, freq=self.freq)
        else:
            obj.index = obj.index._with_freq(self.freq)
```

test_monthly_convention_span (reason="Commented out for more than 3 years. Should this work?") -- I think it (https://github.com/pandas-dev/pandas/blob/e4956ab403846387a435cd7b3a8f36828c23c0c7/pandas/tests/resample/test_period_index.py#L684C1-L696C49) should not work. Even correcting to something like:

def test_monthly_convention_span(self):

    rng = period_range("2000-01", periods=3, freq="M")
    ts = Series(np.arange(3), index=rng)

    # hacky way to get same thing
    exp_index = period_range("2000-01-01", "2000-03-31", freq="D")
    expected = ts.asfreq("D", how="start").reindex(exp_index)
    expected = expected.ffill()

    result = ts.resample("D").ffill()

    tm.assert_series_equal(result, expected)

we have


AssertionError: Attributes of Series are different

Attribute "dtype" are different

which I think is expected (after reindex we have many NaN, changing the type to float64 and this continues after ffill)

And, using the fix that I'm suggesting, we will have:
``` python
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[mean] - AssertionError: Attributes of Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[sem] - AssertionError: Attributes of Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[median] - AssertionError: Attributes of Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[var] - AssertionError: Attributes of Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[std] - AssertionError: Attributes of Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[ohlc] - AttributeError: 'DataFrame' object has no attribute 'dtype'. Did you mean: 'dtypes'?
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[quantile] - AssertionError: Attributes of Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[count] - AssertionError: Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[size] - AssertionError: Series are different
FAILED test_period_index.py::TestPeriodIndex::test_resample_same_freq[nunique] - AssertionError: Series are different

because https://github.com/pandas-dev/pandas/blob/e4956ab403846387a435cd7b3a8f36828c23c0c7/pandas/tests/resample/test_period_index.py#L287C1-L294C1:

    def test_resample_same_freq(self, resample_method):
        # GH12770
        series = Series(range(3), index=period_range(start="2000", periods=3, freq="M"))
        expected = series

        result = getattr(series.resample("M"), resample_method)()
        tm.assert_series_equal(result, expected)

is expecting the wrong behavior pointed out by #53481. I think it's ok to expect the same values using sum, mean, etc... but how could we expected the same values/attributes using methods like std, var, ohlc, nunique, size, count, etc? I didn't find similar test for datetime_index.

pandas-dev / pandas