apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.44k stars 959 forks source link

[Bug] flink cdc database sync will throw an exception when drop table #1700

Closed zhangjun0x01 closed 1 year ago

zhangjun0x01 commented 1 year ago

Search before asking

Paimon version

master

Compute Engine

flink k8s

Minimal reproduce step

  1. start a flink job to sync mysql database to paimon
  2. drop a mysql table
  3. it throw an exception
    Invalid historyRecord, because tableChanges should contain exactly 1 item

What doesn't meet your expectations?

the message is like this:

{
    "source":{
        "file":"mysql-bin.000040",
        "pos":2266343,
        "server_id":1
    },
    "position":{
        "transaction_id":null,
        "ts_sec":1690875697,
        "file":"mysql-bin.000040",
        "pos":2266468,
        "server_id":1
    },
    "databaseName":"db1",
    "ddl":"DROP TABLE `db1`.`t4`",
    "tableChanges":[

    ]
}

The program will throw an exception and be blocked , the subsequent operations cannot be synchronized

Anything else?

No response

Are you willing to submit a PR?

yuzelin commented 1 year ago

Solved by #1702