search

deephaven / deephaven-core

Deephaven Community Core
Other
255 stars 80 forks source link

Additional operator for `update_by` requested #5709

Open lbooker42 opened 4 months ago

lbooker42 commented 4 months ago

To support production use cases, we need the following operators (also found in #4424):

But also needed are the following (supported by pandas / Polars):

chipkent commented 2 months ago

Other count operations like count_null, count_nan, etc. would be useful.

chipkent commented 2 months ago

As we have done in other cases, null values should be ignored, and NaN values are included -- typically resulting in poisoning.

chipkent commented 2 months ago

I looked through Pandas docs and found a few more operations that we should really support:

chipkent commented 2 months ago

Below is an attempt at a more comprehensive and carefully curated list.

As has been the case for other operations:

  1. null values are ignored in calculations.
  2. NaN values are included in calculations. Typically, this means that NaN poisons results, so the operator will return NaN after seeing a NaN.
  3. +0.0 and -0.0 are considered to be the same and equivalent.

Operators have a few different contexts:

  1. agg
  2. update_by cumulative
  3. update_by window / rolling

Missing cumulative operators:

New operators (singleton)

New operators (agg, cumulative, and rolling):

Don't Do Operators? (Present in agg)

These are present in agg, but they may not be worth adding to the other cases until there is demand. They need some discussion.

(?) There will be some debate on if this method should be implemented because of efficiency. (*) May involve some tricky, careful numerics to compute good values. Need to be careful in defining the calculation. (+) Not yet implemented in Numerics.ftl (!) There has been some discussion around these operations with @rcaudy and @chipkent . cum_first/cum_last are the same as first_by/last_by, so there is an argument to not include them. offset is proposed as a way to get a value at a specific index or time offset instead of having a first/last operator. For time offsets, there needs to be a way to disambiguate if there are multiple values with the same time offset. offset would not be supported by agg, but first and last would.

chipkent commented 1 month ago

Details on computing skewness and excess kurtosis can be found at:

We want the sample skewness and sample excess kurtosis. The formulae used by Excel, SAS, etc. have probably been well vetted.

chipkent commented 1 month ago

Details on computing the sample covariance can be found at: