apache / amoro

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
https://amoro.apache.org/
Apache License 2.0
874 stars 290 forks source link

[Improvement]: Separate the shared iceberg-worker-pool for planning and committing #3312

Closed rfyu closed 4 days ago

rfyu commented 3 weeks ago

Search before asking

What would you like to be improved?

Currently, the plan and commit operations share the same iceberg worker thread pool. When a large table with many small files is being planned, it can occupy a large number of threads. This may cause other tables that need to commit to remain in a committing state due to a lack of available worker threads. Therefore, it is necessary to separate them.

How should we improve?

No response

Are you willing to submit PR?

Subtasks

No response

Code of Conduct