alamb
commented
2 years ago https://github.com/apache/arrow-datafusion/issues/2551 might also belong listed in the "other" category
Open andygrove opened 2 years ago
alamb
commented
2 years ago https://github.com/apache/arrow-datafusion/issues/2551 might also belong listed in the "other" category
andygrove
commented
2 years ago apache/arrow-datafusion#2551 might also belong listed in the "other" category
Thanks @alamb. Strictly speaking, we're not blocked on this because we are only using the SQL query planner and the logical optimization rules. However, implementing the physical plan for OFFSET and adding some integration tests would give me greater confidence that the planning and optimization rules for LIMIT and OFFSET are correct, so I will add it to the list.
The purpose of this issue is to track the work that we need to do in the DataFusion project to support moving the dask-sql planner to DataFusion.
High Priority Tech Debt
We need to fix some issues before we can really get started on the main features.
High Priority Features
The SQL Query Planner and Logical Plan need to implement these features. Note that DataFusion does not necessarily need to implement a physical plan for these features, so that reduces the scope of this work.
High Priority Bugs
These are the bugs that we are seeing when attempting to parse all the queries from our benchmark suite.
Ongoing improvements
We do not need these for the benchmark suite but these are features and bugs that we are likely to eventually run into so it makes sense to be proactive and work on these.
Refactoring of DataFusion crates
We currently bring in the full datafusion crate as a dependency, including the physical plans and execution engine. We really should just depend on the features necessary for SQL query planning and logical plan optimization. These are the issues that need to be implemented to achieve that.
Lower Priority Tech Debt / Misc Other Items