The current "execution plan" is close, but not quite correctly adapted for use when describing the steps necessary to perform a computation.
Ideally, we would introduce a physical plan that is more similar to relational query engines, allowing us to leverage existing techniques and creating the options to run on existing systems.
For now, the plan is to introduce these and move execution towards running them directly, and then (separately) work towards compiling queries directly to physical plans.
[x] Introduce serde compatible physical plan structs.
[x] Introduce a "pipeline scheduler" putting schedule information in the physical plans
[ ] Add merge based pipelines (repartition, merge)
[ ] Figure out how to incorporate aggregations
[ ] Connect to "real" sources
[ ] Connect to real sinks
[ ] Convert the existing execution plan to a physical plan.
[ ] Update tests to run (with a flag) using the converted plans
[ ] Introduce a protobuffer encoding for the physical plan. It may be possible to share this with a serialization of logical plans since they will have similar structure. Ideally, this would allow for source/sink extensibility.
407 was a refactoring to move the ScalarValue into a more accessible location for the physical plans. For now, the intention is to use the ScalarValue within the physical plan to represent literal values. In the long term we may want to revisit that and use a better encoding that would be more aligned with logical plans, but can revisit once the basic plumbing is laid out a bit better.
bjchambers
commented
1 year ago
Summary
The current "execution plan" is close, but not quite correctly adapted for use when describing the steps necessary to perform a computation. Ideally, we would introduce a physical plan that is more similar to relational query engines, allowing us to leverage existing techniques and creating the options to run on existing systems.
For now, the plan is to introduce these and move execution towards running them directly, and then (separately) work towards compiling queries directly to physical plans.