search

apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.36k stars 1.2k forks source link

Add `SessionConfig` reference to `ScalarFunctionArgs` #13519

Open alamb opened 3 days ago

alamb commented 3 days ago

Is your feature request related to a problem or challenge?

@Omega359 noted that by adding SessionConfig to the ScalarFunctionArgs unblock several tasks such as

Describe the solution you'd like

Add &SessionConfig to ScalarFunctionArgs, and ideally add a test that shows the config gets through.

Describe alternatives you've considered

No response

Additional context

We should try and get this done before DataFusion 44 is released so it isn't a breaking change

Omega359 commented 3 days ago

Well, small issue. ScalarFunctionArgs is in datafusion-expr which can't use SessionConfig directly since it's in datafusion-execution which depends on datafusion-expr which would cause a circular dependency. We can use ConfigOptions there which would give us access to all config except the opaque extensions in SessionConfig which I think is acceptable.

niebayes commented 3 days ago

Great work!

Omega359 commented 3 days ago

Started delving into this trying to find a good way to impl. Trying to add config_options: ConfigOptions to ScalarFunctionExpr Having lots of fun with eq and hash with SessionConfig and f64 atm.

alamb commented 3 days ago

I wonder if we can use this trait: https://docs.rs/datafusion/latest/datafusion/catalog/trait.Session.html

Omega359 commented 3 days ago

We could but it wouldn't solve the issue. The issue is PhysicalExpr requires implementations to impl Eq and Hash or to have implementations for DynEq and DynHash. That is fine until something like SessionConfig which doesn't is introduced. I am looking at updating the either the 'config_namespace' macro or add explicit implementations for 'ScalarFunctionExpr' to try and impl either of the above and use f64.to_bits() and f64::from_bits(..) to handle the problematic f64's in the config. I think it's possible

jayzhan211 commented 3 days ago

It makes sense to me to move SessionConfig to common or common-runtime crate

Omega359 commented 2 days ago

take

alamb commented 2 days ago

It makes sense to me to move SessionConfig to common or common-runtime crate

I agree

Omega359 commented 2 days ago

I mistyped above - I meant we couldn't use SessionContext, not SessionConfig. Sorry about the confusion. ... or not. I need more caffeine today. PR uses ConfigOptions.