Open alamb opened 1 day ago
BTW here are the flamegraphs:
FYI @Rachelint / @jayzhan211 -- this might be an interesting project
Actually interesting, I am willing to help push it forward
take
I think it will be quite a cool optimization -- specifically checking for equal values can likely be optimized using the inlined prefix
Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/12680
In https://github.com/apache/datafusion/pull/12269 @jayzhan211 made significant improvements to how group values are stored in multi-column aggregations. This requires specialized implementations for different column types
His initial PR has implementations for PrimitiveArray and String/Binary. However it does not have a specialization for
StringView
So that means that queries that group on multiple columns are even faster. This shows up by effectively slowing down some clickbench queries when they are run with StringView:
For example, this query is 10% slower with StringView
Describe the solution you'd like
I would like to make this (and similar) query faster when string view is enabled :
Note this is grouping by 2 columns
Here is how to reproduce the issue
Step 1. Get
hits.parquet
usingbench.sh
:Step 2: Prepare a script with reproducer query:
Step 3: Run query
set datafusion.execution.parquet.schema_force_view_types = true;
--> Elapsed 0.688 seconds.set datafusion.execution.parquet.schema_force_view_types = false;
--> Elapsed 0.565 seconds.Describe alternatives you've considered
I suggest implementing something like
ByteViewGroupValueBuilder
following the model ofByteGroupValueBuilder
https://github.com/apache/datafusion/blob/6f8c74ca873fe015b2e7ff4eeb2f76bf21ae9d0e/datafusion/physical-plan/src/aggregates/group_values/group_column.rs#L177
The in progress values would be
u128
s and some buffers (maybe 2MB?)implementing
equal_to
can take advantage of the inlined prefix optimization (aka compare the prefix inlined in the u128 and only check the value in the buffer if that is already equal)Additional context
No response