apache / amoro

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
https://amoro.apache.org/
Apache License 2.0
874 stars 290 forks source link

[AMORO-3317]: Move optimizing planer and scan to iceberg/mixed-format module #3314

Closed baiyangtx closed 1 week ago

baiyangtx commented 3 weeks ago

Why are the changes needed?

close #3317

Currently, the Optimizing Plan has become a bottleneck of AMS. A feasible solution is to delegate the Optimizing Plan to the Optimizer for execution.

For this, some prerequisite refactoring work is required.

This PR moves the code of the Optimizing Plan from the AMS module to the iceberg/mixed-format module, and the relevant logic can be called by the optimizer in the future.

Brief change log

How was this patch tested?

Documentation

baiyangtx commented 1 week ago

LGTM. Question: After this refactoring, can Spark customize procedures to trigger optimization? (only depend on the plan and scan logical)

  1. Committer still in ams-module
  2. These refactorings only involve core-level code, and the engine still needs to adapt.