StarRocks / starrocks-connector-for-apache-flink

Apache License 2.0
189 stars 152 forks source link

[Enhancement] Prepare/commit transaction stream load asynchronously #328

Closed banmoy closed 8 months ago

banmoy commented 8 months ago

What type of PR is this:

Which issues of this PR fixes :

Fixes # For transaction stream load at at-least-once mode, only load phase is executed asynchronously in a thread pool whose size is controlled by sink.io.thread-count , but the prepare/commit phase is executed in the single thread used for schedule, and it can block the schedule thread if prepare/commit takes much time. For multiple tables case, this can prevent to schedule other tables to load/prepare/commit, and lead to performance issue. So we make the prepare/commit also asynchronous.

Problem Summary(Required) :

Checklist: