[Umbrella] CDC DDL Sync Design(Zeta)

[hailin0]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I had searched in the issues and found no similar issues.

Describe the proposal

Backgroud

Currently, we have support for data change capture(CDC #3175), but no further design for schema evolution.

• MySQL
• Oracle
• Postgres
• SQlServer
• ...

And as CDC data synchronization, I think we need to support schema evolution(DDL) as a feature, and I want to hear from you all how you think it can be implemented in SeaTunnel.

Motivation

• Support read database ddl sql (mysql/oracle/...)
• Support ddl sync to any sink（Standardized parse ddl to event）
• Support pause and resume ddl at any time
• Support automatic processing of old data and new data switche

Overall Design

Basic flow

• Depends on checkpoint to push ddl: reader -> writer -> committer

Phase1 - Before Change

• Data flow mixed with structure flow
• DDL is bound to database syntax

Phase2 - Starting Change

• Parsed ddl into schema events

Phase3 - Splitting data flow and structure flow

• Insert checkpoint signals before and after ddl
  • schema-change-before signals
  • schema-change-after signals

Phase4 - Handling schema-change-before signal

• Stop source read data
• Waiting for all old data refresh to sink storage

Phase5 - Execute ddl into source & sink

• Stop checkpoint manager

Phase6 - Handling schema-change-after signal

Phase7 - Completed

• Restart checkpoint manager
• Restart read data for source

Task list

• [x] https://github.com/apache/seatunnel/pull/5125
• [x] https://github.com/apache/seatunnel/pull/7463
• [x] https://github.com/apache/seatunnel/pull/6929
• [x] https://github.com/apache/seatunnel/pull/7908

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[hailin0]

link https://github.com/apache/seatunnel/issues/3175

[Carl-Zhou-CN]

Niubility awesome

[dailai]

Niubility awesome

+1

[Carl-Zhou-CN]

Niubility awesome

+1

Ha ha, you learned