[EPIC] Substrait: Add producer and consumer for physical plans

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do. I would like to use substrait with physical plans. I plan on having an initial PR up this weekend.

Describe the solution you'd like

• Create new consumer and producer for physical plans

Describe alternatives you've considered

Additional context Substrait to DataFusion's logical plan is tracked at https://github.com/apache/arrow-datafusion/issues/8149

Tasks:

• [x] https://github.com/apache/arrow-datafusion/pull/5176
• [ ] https://github.com/apache/arrow-datafusion/issues/9299
• [ ] https://github.com/apache/arrow-datafusion/issues/9347
• [ ] https://github.com/apache/arrow-datafusion/issues/8698
• [ ] https://github.com/apache/arrow-datafusion/issues/8361
• [x] https://github.com/apache/datafusion/issues/10815
• [x] https://github.com/apache/datafusion/issues/10817
• [ ] https://github.com/apache/datafusion/issues/9727
• [ ] https://github.com/apache/datafusion/issues/9347
• [ ] https://github.com/apache/datafusion/issues/9299
• [ ] https://github.com/apache/datafusion/issues/10710
• [ ] https://github.com/apache/datafusion/issues/10864

[andygrove]

@waynexia @nseekhao fyi

[waynexia]

What is the expected behavior for converting "LogicalPlan -> Substrait -> PhysicalPlan" or "PhysicalPlan -> Substrait -> LogicalPlan"? Or is it allowed?

[alamb]

I renamed this ticket to be an epic and started collecting tasks needed for better support