OmnisciDB UDF support allows users to define and register custom functions (UDFs) to OmnisciDB server so that these can be used withing SQL queries to the server. The UDFs implementation is given as an LLVM IR module string that is linked to the server SQL engine in runtime. OmnisciDB server defines Thrift API for managing the UDFs.
The RBC project provides a Python package rbc that allows users to define and register the UDFs using a Python function definition. The rbc package uses numba for LLVM IR module string generation and registers the UDFs to OmnisciDB server via its Thrift API.
Features
[x] Retrieve device (CPU, GPU) information from the server required for LLVM IR generation.
[ ] Generalized calls within UDFs to external C/C++ libraries (Stan, XGBoost, CUML, etc).
Basic idea: calling a function from an external library from a UDF definition boils down to allowing undefined symbols in the LLVM IR module that are resolved on the server-side. The server must be linked against the external library and provide this information to RBC via device information retrieval Thrift API.
Numba is currently undergoing a refactoring process (PR https://github.com/numba/numba/pull/5197, etc) that will require the corresponding adjustments in RBC when numba 0.49 is released.
OmnisciDB Thrift API changes (e.g. adding new types) requires syncing rbc accordingly. Solution: include type information to device information retrieval end-points.
Background
OmnisciDB UDF support allows users to define and register custom functions (UDFs) to OmnisciDB server so that these can be used withing SQL queries to the server. The UDFs implementation is given as an LLVM IR module string that is linked to the server SQL engine in runtime. OmnisciDB server defines Thrift API for managing the UDFs.
The RBC project provides a Python package
rbc
that allows users to define and register the UDFs using a Python function definition. Therbc
package usesnumba
for LLVM IR module string generation and registers the UDFs to OmnisciDB server via its Thrift API.Features
Work-in-progress
Work planned
Possible breakers/blockers