apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
5.49k stars 1.02k forks source link

Add subquery support in `datafusion-proto` #2640

Open andygrove opened 2 years ago

andygrove commented 2 years ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. Once https://github.com/apache/arrow-datafusion/pull/2639 is merged we should add support for serializing plans that contain subquery expressions.

This will likely require breaking changes to the API because we currently have one API for Expr that optionally accepts a FunctionRegistry and one API for LogicalPlan that optionally accepts a LogicalExtensionCodec. With subquery support we have plans referencing expressions and expressions referencing plans, so these APIs will need to be combined somehow.

Here is a suggestion for the new API from the discussion at https://github.com/apache/arrow-datafusion/pull/2639#discussion_r884249631

let bytes = PlanSerializer::new()
    .with_extension_codec(&extension_codec)
   .serialize()?;

Describe the solution you'd like TBD

Describe alternatives you've considered None

Additional context None

liukun4515 commented 2 years ago

feature of the subquery is needed in my system, I can figure out this issue. @andygrove

liukun4515 commented 2 years ago

also with that: https://github.com/apache/arrow-datafusion/issues/2338 and https://github.com/andygrove/arrow-datafusion/blob/39c444835ddcb38cf1efeb4414fbb8ecd49ac020/datafusion/proto/src/to_proto.rs#L710

alamb commented 2 years ago

When we implement this feature, it might be worth considering a slightly more encapsulated API for serializing / deserializing plans, as described in https://github.com/apache/arrow-datafusion/pull/2639#discussion_r884249631

andygrove commented 2 years ago

I have a PR up (https://github.com/apache/arrow-datafusion/pull/2670) with a proposal for the API part

alamb commented 1 year ago

Possibly duplicated by https://github.com/apache/arrow-datafusion/issues/2646