deephaven / deephaven-core

Deephaven Community Core
Other
257 stars 80 forks source link

Improve `AggFormula` performance by re-using `GroupByChunkedOperator` rather than re-creating new operators. #6363

Open lbooker42 opened 1 week ago

lbooker42 commented 1 week ago

Initial provision is made to re-use a shared GroupByChunkedOperator in FormulaMultiColumnChunkedOperator and FormulaChunkedOperator but more work is needed before this can be fully used.

1) We need to optimize the operator order in AggregationOptimizer to look for GroupByChunkedOperator instances from group aggs or that may be created for other formulas. 2) For FormulaMultiColumnChunkedOperator, we need to update GroupByChunkedOperator to keep two sets of output columns: public columns for requested GroupBy aggs and "hidden" ones only exposed to internal operators.