Open alamb opened 11 months ago
One observation here is that min
and max
on strings is not that common of an operation from what it seems -- grouping on strings is more common.
Maybe there is some binary usecase where it is important (e.g. embeddings π€ )
Is your feature request related to a problem or challenge?
https://github.com/apache/arrow-datafusion/pull/6904 introduces some fancy new hashing and ways to implement aggregates
min/max for strings (
StringArray
/LargeStringArray
, etc) now uses the slowerAccumulator
implementation which could be made much fasterDescribe the solution you'd like
I would like to implement a fast
GroupsAccumulator
for Min/MaxDescribe alternatives you've considered
here is one potential way to implement it:
We could store the current minimum for all groups in the same Rows π€ and track an index into that Rows for the current minimum for each group.
This would require an extra copy of the input values, but it could probably be vectorized pretty well, as shown in the following diagram.
Sorry what I meant was something like the following where the accumulator only stored the current minimum values.
This approach would potentially end up with
min_storage
being full of "garbage" if many batches had new minumums, but I think we could heuristically "compact"min_storage
(if it had2*num_groups
, for example) if it got too largeSee https://github.com/apache/arrow-datafusion/pull/6800#issuecomment-1622290981 for more details
Additional context
No response