NaN filtering

[kaanolgu]

Can NaN-type values be filtered out by our existing filtering capabilities @pineapple-cat ? If not, we'll need it.

[pineapple-cat]

I haven't tested whether a filter like ["col_name", "!=", null] would work as expected, but what you're asking for is not currently intentionally implemented. I've made a note of this before though, and it would be quite simple to add a flag to the config to signify that any rows with NaN values should be dropped. We could situate it like this:

y_axis:
  value: "y_axis_col"
  units:
    column: "unit_col"
  drop_nan: True

And then add something like this to the post-processing code:

if config["y_axis"].get("drop_nan"):
  df.dropna(subset=[config["y_axis"]["value"]], inplace=True)

We could also use this to drop data that has been scaled with itself if we add this line right after scaling in transform_axis:

df[axis["value"]].replace(to_replace=1, value=np.NaN, inplace=True)

[ilectra]

The method to drop them looks fine, but I think notionally it makes sense more as a filter than a property of the y-values. Maybe we could have some pre-set filters, like the flag you suggested, for ease of use? What do you think @kaanolgu , as our guinea pig user? :) Also, a flag to drop the self-scaled value could be useful, but I wouldn't like to entangle it with the drop_nan flag.

[ilectra]

I take the above back. Let's treat it as a property of the y-values, set at the config file, for now. If we get more cases for "pre-set filters" in the future, then we move it to the filters.