Flink CDC 3.1.0 Plan

[leonardBang]

Motivation

This is an umbrella issue for Flink CDC 3.1 version

Solution

[module] flink-cdc-common

• [x] #2857
• [x] #2936
• [x] #2943

[module] flink-cdc-cli

• [x] #2934
• [x] #2940

[module] flink-cdc-composer

• [x] #2932
• [x] #2882
• [x] #2854

[module] flink-cdc-connect/flink-cdc-source-connectors

• [x] #2869
• [x] #1747
• [x] #2867
• [x] #1152
• [x] #2941

[module] flink-cdc-connect/flink-cdc-pipeline-connectors

• [x] #2691
• [x] #2856

[module] flink-cdc-e2e-tests

• [x] #2859

[module] docs

• [x] #2935
• [x] #2940

Bug fix

• [x] #2865
• [x] #2853
• [x] #2966
• [x] #2905

Design docs

https://docs.google.com/document/d/1sC045l08hqZ8C9bVxeZdH5fY5A4Ljz__f5SqZ2Z8qrM/edit?usp=sharing

[lvyanquan]

Please add this Bug fix to the plan: Got NoSuchMethodError when running with Flink 1.17.1 and before. https://github.com/ververica/flink-cdc-connectors/issues/2943

[ChenShuai1981]

please support kubernetes deployment and auto create database

[ChenShuai1981]

please support pipeline with one source and many sinks

[miaogr86]

What are the ways to reduce memory usage during the snapshot phase? https://github.com/ververica/flink-cdc-connectors/pull/1629 https://github.com/ververica/flink-cdc-connectors/issues/2700

[ChenShuai1981]

Support add customized type mapping like paimon does, for example, tinyint1-not-bool and bigint-unsigned-to-bigint https://paimon.apache.org/docs/master/cdc-ingestion/mysql-cdc/

[leonardBang]

Support add customized type mapping like paimon does, for example, tinyint1-not-bool and bigint-unsigned-to-bigint https://paimon.apache.org/docs/master/cdc-ingestion/mysql-cdc/

+1, could you open a ticket for this ?

[xiaolianghu]

Look forward to the 3.1 release soon

[klion26]

hi @leonardBang could #2315 be fixed in 3.1.0 ?

Seems the PR has been filed a long time ago, the author active on it, and it can fix many problems.

[komiblog]

Please merge this pr #2494 to fix the MariaDB connector bug, thank you very much！

[maver1ck]

Do you have plans to include general JDBC sink ? (not only starrocks ?)

[ChenShuai1981]

Support add kafka source with debezium and canal json format

[viethung2281996]

hi @leonardBang , do you have plan or roadmap for module flink-cdc-pipeline-connectors to support pipeline with MongoDB source !

[leonardBang]

hi @leonardBang , do you have plan or roadmap for module flink-cdc-pipeline-connectors to support pipeline with MongoDB source !

Yes, it's on the plan but maybe we consider it in next version

[melin]

support kafka pipeline data source connector

Sometimes cdc data needs to be written to a different data source storage and needs to be written to kafka first. It is then forwarded to different databases or data lakes in real time。

[ChenShuai1981]

How to handle different primary keys need between mysql and doris? For example, in mysql table we always use auto-incremental id column as primary key and device_id as biz unique key, while in doris table, considering query performance by device_id, we choose device_id + id columns as primary key.

[lvyanquan]

How to handle different primary keys need between mysql and doris? For example, in mysql table we always use auto-incremental id column as primary key and device_id as biz unique key, while in doris table, considering query performance by device_id, we choose device_id + id columns as primary key.

@ChenShuai1981 this https://github.com/apache/flink-cdc/pull/2937 pr provide the ability to modify primary key definition.

[hyr666]

when did it support OB for Oracle(Enterprise Edition)?

[PatrickRen]

Closing this issue as it has been migrated to Apache Jira.

[xiaolianghu]

Does release 3.1.0 support the following features? Thanks! 【Support dynamically table addition in flink-cdc-base #1152】 @leonardBang