Open wbeardall opened 5 months ago
thanks @wbeardall for the request! seems reasonable, will take a look
I've submitted a PR for how I'd go about implementing this. Let me know any thoughts!
similar to in the builtin ewma, to allow for NaN or null values to be ignored during calculation
Are you sure this is what the ewm_mean
one does?
In [15]: s = pl.Series([1.1, 2.5, 2.6, 2.1, float('nan'), 5.1])
In [16]: s.ewm_mean(alpha=.1, ignore_nulls=True)
Out[16]:
shape: (6,)
Series: '' [f64]
[
1.1
1.836842
2.11845
2.113085
NaN
NaN
]
Looks like NaN values still propagate there?
Which looks correct to me - Polars (unlike pandas) generally distinguishes NaN and null
I think you might be right here; in honesty, I'm a very recent convert from Pandas, and may have mistaken the design pattern in pl.ewm_mean
. The main motivation for this issue was to create a feature by which you can prevent NaN
propagation in time series data, similar to how Pandas handles the presence of NaN
elements with their ignore_na
flag (below); this might not necessarily be the same as the ignore_nulls
feature of pl.ewm_mean
. Seems like this is the same question as yours last night about the need to distinguish between NaN
and null
values.
>>> import pandas as pd
>>> s = pd.Series([1.1, 2.5, 2.6, 2.1, float('nan'), 5.1])
>>> s.ewm(alpha=.1, ignore_na=True, min_periods=1).mean()
0 1.100000
1 1.836842
2 2.118450
3 2.113085
4 2.113085
5 2.842473
dtype: float64
My particular use-cases, and the PR that I submitted last night, are focused on the ignore_na
case, which I would appreciate being a feature; is it worth adding ignore_nulls
as another feature under the same or another issue?
Perhaps it is better to propagate NaNs, whilst having null
values behave as initially written in the PR, and communicate to users that this is the mechanism by which they should enable propagation past such values? e.g.
>>> import polars as pl
>>> s = pl.Series([1.1, 2.5, 2.6, 2.1, float('nan'), 5.1])
>>> s.fill_nan(None).ewm_mean(alpha=.1, ignore_nulls=True)
shape: (6,)
Series: '' [f64]
[
1.1
1.836842
2.11845
2.113085
2.113085
2.842473
]
yeah doing s.fill_nan(None)
first feels like idiomatic Polars
I've pushed an implementation for the above, as well as improving robustness. In the previous version, if a series started with a null value, the kernel would panic, as it would attempt to call .unwrap()
on said null; let me know thoughts
Currently, if there are any NaN values in the value column passed to
xdt.ewma_by_time
, then all following values in the output are NaN (see snippet). It would be great if there was nignore_nulls
flag, similar to in the builtinewma
, to allow for NaN or null values to be ignored during calculation, to prevent this. In this case, the presence or absence of a row containing Null or NaN should have no effect on subsequent rows; i.e. theewma
-ed output of the final row of the two following tables should be identical.Reproducible snippet