Open findepi opened 2 days ago
cc @alamb, @andygrove, @jayzhan211, @ozankabak, @notfilippo, @comphead
FYI: I am doing some experiments how this could look like
I think the idea of making it easier to write functions that include specialized implementations for different types is a great idea. This would likely both make our code faster (more specializations, more uniformly handle constants) as well as reduce the replication (there are several different patterns for dispatch / specialization now -- consolidating would be really nice)
Currently DataFusion functions are singletons plugged into the execution engine. They have no way to store and reuse buffers or compiled regular expressions, etc.
here is some prior work on the precompiled regular expressions: https://github.com/apache/datafusion/issues/11146 (note compiling the regex hasn't ever shown up in any performance trace I did, probably because the cost of actually matching is so much bigger than the cost of compiling and the cost of compiling is spread over 8192 rows
Thanks @findepi I think this process go through iterations, and easier than was before but still far from perfect.
The ScalarUDFImpl common trait is already a huge help, but for example dynamic typing, like MAX/MIN/GREATEST
when return type depends on input is gonna be challenging, in Trino it seems also implemented in runtime rather than static.
Rust can benefit from macros and generate functions with all possible output types combinations, but this is just 1 problem.
I'm totally down if we can make it easier
Just to be clear, what I am imagining comes out of this work is:
ScalarUDFImpl
Is your feature request related to a problem or challenge?
Verbosity
Currently implementing a scalar function is a pretty involved process. For example a simple function calculating greatest common divisor looks like this (a lot of detail omitted for brevity):
This function is still "relatively simple" because:
return_type
and again ininvoke
x
or constanty
parametersmake_scalar_function
helper expands scalar constant argument into an array, thusgcd(x, 5)
is less efficient thangcd(x, y)
because it first needs to allocate a temporary buffer for5
s repeated times the batch lengthgcd
on partitioning keys, but this is just an exampleIt should be possible to express functions using simple row-level operations, because in query engines most function logic is structured like that anyway (
compute_gcd
in the example).Local data structures
Currently DataFusion functions are singletons plugged into the execution engine. They have no way to store and reuse buffers or compiled regular expressions, etc. Thread local storage is not an option because DataFusion, when embedded, does not control creation of application threads.
Describe the solution you'd like
Simple
It should be possible to separate function logic from the mechanics. Exemplary GCD function needs to provide
fn compute_gcd(x: i64, y: i64) -> Result<i64>
and the rest of boilerplate should not be hand-written for every function separately.It should be possible to implement a function that accepts string values, without having to deal with the 5 different runtime representations that can carry string values:
StringArray, LargeStringArray, StringViewArray, DictionaryArray<Key>, RunArray<Key>
(maybe more than 5 because they are recursive in theory: canDictionaryArray
contain aDictionaryArray
? can it containRunArray
?)Focused
Because SQL is statically typed, it is necessary to select function overload during planning time, so that return type is also known (the
return_type
function). Theinvoke
function needs to implement same logic again. This process should be less error-prone: once the function is bound to the query call site, its signature is known and the implementation should not need to do type checks.Performant / Efficient
It should be the compiler's / framework's job to provide vectorization, without hand-writing same logic in every function separately.
It should be the compiler's / framework's to do null checks, providing efficient tight loops for the all-non-null case without having to write such logic in every function separately.
It should be possible to write efficient functions that need thread local data structures, for example for regular expressions, without having to use thread locals and/or shared pools which introduce contention.
Describe alternatives you've considered
However, direct use of the library is not possible because
The library could still be part of the solution, but doesn't have to be and it's a non-goal to use a particular library.
Additional context