search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.44k stars 959 forks source link

[cdc] Add flink execution mode check for cdc table sync action. #4378

Closed LinMingQiang closed 3 weeks ago

LinMingQiang commented 1 month ago

Purpose

Linked issue: close #xxx

Flink-cdc do not support batch mode, including flink cdc snapshot scan mode . Although it is a bounded source.

https://issues.apache.org/jira/browse/FLINK-36280

https://github.com/apache/flink-cdc/pull/2901

if use set RuntimeExecutionMode = Batch. job will be never finish and never fail.

image

so , we need to check the flink execution mode.

Tests

API and Format

Documentation

LinMingQiang commented 1 month ago

If this change is acceptable, I will complete the other actions / mongodb/Postgres.

wwj6591812 commented 1 month ago

Do we need to handle RuntimeExecutionMode.AUTOMATIC?

LinMingQiang commented 1 month ago

Do we need to handle RuntimeExecutionMode.AUTOMATIC?

yes need too, may be we can make STREAMING mode as default and ignore the user's settings.

LinMingQiang commented 4 weeks ago

@yuzelin Help me check this pr? tks.