search

elixir-explorer / explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir
https://hexdocs.pm/explorer
MIT License
1.12k stars 123 forks source link

Series nil vs NaN #1010

Open kylewhite21 opened 4 weeks ago

kylewhite21 commented 4 weeks ago

Are there any recommendations on how to deal with Series NaN values? I'd like to use fill_missing/2 or coalesce, but they seem to ignore NaN. Even if it were possible to explicitly convert or cast a NaN to nil first.

alias Explorer.Series, as: S

s1 = S.from_list([2, 0])
s2 = S.from_list([1, 0])
S.divide(s1, s2) |> S.fill_missing(0.0)

#Explorer.Series<
  Polars[2]
  f64 [2.0, NaN]
>

s = S.from_list([1, nil])
S.fill_missing(s, 0)

#Explorer.Series<
  Polars[2]
  s64 [1, 0]
>
josevalim commented 4 weeks ago

You can probably do this: S.select(S.is_nan(s), 0, s) although we may consider adding a fill_nan for convenience.

kylewhite21 commented 4 weeks ago

Ok, thanks for the quick reply as usual. I'm good with this. You're welcome to close it unless you want to use it to track work for fill_nan.