Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)
Currently Ballista does not support rust UDF, which makes it hard to process data using some custom function or with external libraries.
Describe the solution you'd like
A clear and concise description of what you want to happen.
rust has never guaranteed a stable ABI (i.e. memory layout), therefor the fields in the UDF class in the plugin maybe interpreted incorrectly, e.g. ColumnarValue or Signature, etc.
in practice same rustc version + same optimization level gives the same ABI (i.e. memory layout for the class). this suggest the plugin must be complied with the exact same rustc and compiler flags.
or alternatively we could use a stable api library such as abi_stable or stabby marks all UDF related class
Reconstruct a UDF function a function with Arrow data as parameter and Arrow data as return type
we lost lots of flexibility provided by rust ScalarUdfImpl, such as change the signature by value, or provide specialized code path to use the ColumnarValue::Scalar
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
alternatively, we could build a custom ballista executor each time we want to add or modify a UDF and deploy it.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem or challenge? Please describe what you are trying to do. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and why for this feature, in addition to the what)
Currently Ballista does not support rust UDF, which makes it hard to process data using some custom function or with external libraries.
Describe the solution you'd like A clear and concise description of what you want to happen.
There are two possible Solutions
Register the UDF directly
We load rust dynamic library into the executor (similar to this PR: https://github.com/apache/arrow-datafusion/pull/1881, and we have partial code here https://github.com/apache/arrow-ballista/tree/main/ballista/core/src/plugin) and register the UDF directly to the DataFusion.
issues:
ColumnarValueorSignature, etc.abi_stableorstabbymarks all UDF related classReconstruct a UDF function a function with Arrow data as parameter and Arrow data as return type
we can only load a function that use Arrow data as parameter and returns, since this is memory layout stable (e.g. using https://arrow.apache.org/rust/arrow/ffi/struct.FFI_ArrowArray.html). we could pass the signature using a serialized string or something.
similar to this: https://github.com/apache/arrow-datafusion-python/blob/main/src/udf.rs
issues:
ScalarUdfImpl, such as change the signature by value, or provide specialized code path to use theColumnarValue::ScalarDescribe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
alternatively, we could build a custom ballista executor each time we want to add or modify a UDF and deploy it.
Additional context Add any other context or screenshots about the feature request here.