pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
29.69k stars 1.9k forks source link

add strategy="mode" for fill_null #16585

Open zippeurfou opened 4 months ago

zippeurfou commented 4 months ago

Description

This is a small feature request to add the strategy "mode" for fill_null. This is especially useful for column that are string. Doing it via code is straighforward. If not a feature request maybe the doc could provide an example with this instead of using it the median.

s-banach commented 4 months ago

What if there is not a unique mode?

zippeurfou commented 4 months ago

This is a good point. https://docs.pola.rs/py-polars/html/reference/series/api/polars.Series.mode.html do mention the possibility of having multiples values. I guess this would involve having a method to tie break them which could be none or random for example. This will then be a feature request for mode.

ritchie46 commented 4 months ago

You can do that with a when -> then -> otherwise:

foo = pl.col("foo")
pl.when(foo.is_null()).then(foo.mode().first())

And fill_null also accepts expressions:

foo.fill_null(foo.mode().first())
zippeurfou commented 4 months ago

Thanks @ritchie46, I was not mentioning it is not possible. I actually use the fill_null that you mentioned in your example but I was referring that it could be good to either add this example in the doc or as a strategy option for the method arguments in https://github.com/pola-rs/polars/blob/py-0.20.30/py-polars/polars/dataframe/frame.py#L7202 as this is pretty helpful for string and the other alternatives ({None, ‘forward’, ‘backward’, ‘min’, ‘max’, ‘mean’, ‘zero’, ‘one’}) do not help in the case of string.

cmdlineluser commented 4 months ago

Someone did request this previously https://github.com/pola-rs/polars/issues/15007 but they closed the issue.

(So it seems like a useful doc example at the very least?)

.mode() does counts nulls

>>> pl.Series([1, None]).mode()
shape: (2,)
Series: '' [i64]
[
    1
    null
]
>>> pl.Series([1, None, None]).mode()
shape: (1,)
Series: '' [i64]
[
    null
]

I've not seen it mentioned before, so I'm not sure if this is intended or not - but currently you'd need to drop them.

For comparison, pandas drops them by default:

>>> pd.Series([1, None, None]).mode()
0    1.0