enso-org / enso

Enso Analytics is a self-service data prep and analysis platform designed for data teams.
https://ensoanalytics.com
Apache License 2.0
7.38k stars 323 forks source link

`Vector.filter` by `Filter_Condition` does not work well with `Nothing` #8549

Closed radeusgd closed 11 months ago

radeusgd commented 11 months ago

image

As we can see - in Table.filter this works well, because all of our vectorized table operations have special handling for NULLs (Nothing) in the columns.

However, Vector.filter just delegates to raw predicates like .is_finite or (<) which are not defined for Nothing and thus fail.

We should improve Filter_Condition.to_predicate to add special handling for Nothing to allow better filtering of vectors containing nulls.

Note that if the user uses a raw lambda predicate in Vector.filter, i.e. [1, Nothing].filter (> 0) - this is not something in scope of this ticket - the predicate will fail because it expects Number and gets Nothing. This is expected behaviour and is OK because custom predicates are mostly meant for more advanced users. Filter_Condition, however, should be more universal.

enso-bot[bot] commented 11 months ago

Radosław Waśko reports a new STANDUP for yesterday (2023-12-20):

Progress: Pulling the PRs thru reviews and CI. Prepared the PR for handling of nulls in Vector.filter by Filter_Condition. Researched some bug within it. AoC. It should be finished by 2023-12-20.

Next Day: Next day I will be working on the #8555 task. Pick up next task. Get the PRs merged. Finish an AoC task I was trying to do.