search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.3k stars 3.48k forks source link

[Gandiva][UDF] Solutions to register new UDFs dynamically without checking it into arrow repo. #24502

Open asfimport opened 4 years ago

asfimport commented 4 years ago

Hi there,

Recently I'm studying on gandiva and trying to add some UDF. I noted that it's needed to check the UDF implementation into the arrow repo, register the UDF and then build the UDF into precompiled_bitcode lib, right? I'm just wandering that is it possible to register new UDFs dynamically? Say I have the UDF implementation code locally which is not built into the gandiva lib yet, am I able to call some function or other solutions provided by gandiva officially to register and implement it. Thanks in advance.

Reporter: ZMZ91 / @ZMZ91

Note: This issue was originally created as ARROW-8313. Please see the migration documentation for further details.

asfimport commented 4 years ago

Pindikura Ravindra / @pravindra: This functionality is not supported yet and I'm not aware of anyone working on it.   However, I think it shouldn't be very hard to support this. There are two places where gandiva interacts with functions :

  1.  Function registry ** the gandiva code looks for fns based on a signature (includes name & input types) in a function registry. The lookup function returns the name of the pre-compiled function ** This can be enhanced to support multiple registries - allow for registration of additional registried in addition to the static one.
  2. Loading pre-compiled modules ** At the time of building the module, the engine loads the pre-compiled IR. This will need to also load other registered libraries (either as a string or a file).      And, we will need to come up with some scheme to avoid name conflicts - both the names in the registry & the names of the functions.
asfimport commented 4 years ago

ZMZ91 / @ZMZ91: Thanks @pravindra. So do you guys have any plan to support it? Or any way that I can contribute it?

asfimport commented 4 years ago

Praveen Kumar / @praveenbingo: @ZMZ91 As of now, we do not plan to actively work on adding this support. Please go ahead if you want to implement this, we will help out in case of road  blocks.

asfimport commented 4 years ago

ZMZ91 / @ZMZ91: Thanks for your info @praveenbingo.