It seems that whenever the polars-plan crate resolves the type information of an FFI call, it looks for the symbol __polars_field_{}, where {} is the name of the Rust function under #[polars_expr(...)], see this LoC.
The function that is actually exposed for this field resolution is defined here for expressions of the pattern #[polars_expr(type_func=...)], and here for expressions of the pattern #[polars_expr(output_type=...)].
In the latter case, as fn_name is the function name itself (i.e. haversine in this case), get_field_name returns the correct symbol - in the former, fn_name is instead the name of the type resolution functionhaversine_output, meaning the symbol exposed in the FFI is instead __polars_field_haversine_output, which is unknown to polars-plan.
We can verify this by inspecting the .so file created by pyo3:
$ nm -D ./expression_lib/expression_lib/expression_lib.cpython-310-x86_64-linux-gnu.so | grep " T"
00000000001d0eb0 T __polars_field_hamming_distance
00000000001d14f0 T __polars_field_haversine_output
00000000001d0890 T __polars_field_jaccard_similarity
00000000001d0340 T __polars_field_pig_latinnify
00000000001d1110 T hamming_distance
00000000001d1710 T haversine
00000000001d0af0 T jaccard_similarity
00000000001d05a0 T pig_latinnify
If the fix is simply changing the symbol here, I have implemented this in this PR.
I haven't been able to get this feature to work locally, based on the example - note that the
haversine
expression isn't used in the testrun.py
file.When I attempt to use the
haversine
expression, I get the following errorMinimal reproduction
This modification to
./example/derive_expression/run.py
returns the above error, everything else unchanged.Potential fix
It seems that whenever the
polars-plan
crate resolves the type information of an FFI call, it looks for the symbol__polars_field_{}
, where{}
is the name of the Rust function under#[polars_expr(...)]
, see this LoC.The function that is actually exposed for this field resolution is defined here for expressions of the pattern
#[polars_expr(type_func=...)]
, and here for expressions of the pattern#[polars_expr(output_type=...)]
.In the latter case, as
fn_name
is the function name itself (i.e.haversine
in this case),get_field_name
returns the correct symbol - in the former,fn_name
is instead the name of the type resolution functionhaversine_output
, meaning the symbol exposed in the FFI is instead__polars_field_haversine_output
, which is unknown topolars-plan
.We can verify this by inspecting the
.so
file created bypyo3
:If the fix is simply changing the symbol here, I have implemented this in this PR.