ENH: Pandas ewm with fixed window / offset

[nrcjea001]

Problem & Proposed feature Pandas ewm function works similar to the pandas expanding function in that it rolls over the whole dataframe. In my case however, I need to specify a fixed window or offset over which the ewm function is applied. In other words, an ewm function with a cutoff that would work similar to the 'window' parameter in the pandas rolling function.

Alternatives considered My workaround to the problem is as follows (however extremely inefficient):

df = pd.DataFrame({'a': np.random.randint(5, size=24),
                   'b': ["S", "A"] * 12,
                   'c': pd.date_range(start='1/1/2018', end='12/12/2018', freq='15D')})

df.groupby('b').rolling('60d', on='c')['a'].apply(lambda x: x.ewm(halflife='15d', times=x.index).mean().tail(1))

In the example above, I am trying to do an exponentially-weighted moving average, where decay is specified in terms of halflife on a datetime column. The rolling apply ensures that the maximum offset is '60d' which is the feature I propose to be added directly in the ewm function.

So in summary, I propose a window parameter to be introduced to the ewm function which limits the ewm calculation to a fixed window or offset.

[FitzHoo]

When will the feature be added to ewm function, please?

[jreback]

pandas is all volunteer - community PRs are how things get added

this issue likely needs specification of an efficient way of doing this first

[jmg-duarte]

I've implemented a similar feature.

The use case was similar: we need to calculate the EWMA in a "forgetful" manner, where only the values in the window are taken into account.

We perform the calculations in an incremental manner and adapted the formula from adjust=True by removing the weight of the "dropped" value (i.e. the value being removed).

[yongqiang-zhao]

@jmg-duarte Hi, could you kindly please provide details about how to use the feature? Really appreciate your help.

[jmg-duarte]

@yongqiang-zhao sadly I've since moved on to another company and no longer have access to that code.

Best I can do now is provide some resources:

• https://github.com/pandas-dev/pandas/blob/219bc5ecf872ed8527c0aef0b82167b051612c51/pandas/core/window/online.py#L37-L84

• https://github.com/pandas-dev/pandas/blob/fe48704835323c140846d1bde5e1387aa0cac3d4/pandas/algos.pyx#L967-L1023

• https://towardsdatascience.com/time-series-from-scratch-exponentially-weighted-moving-averages-ewma-theory-and-implementation-607661d574fe

• https://stackoverflow.com/questions/42869495/numpy-version-of-exponential-weighted-moving-average-equivalent-to-pandas-ewm

I hope this helps!

[yongqiang-zhao]

@jmg-duarte Thank you very much for your kind help.