apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.78k stars 1.74k forks source link

[Bug] [MySQL CDC] Cannot capture change data #7082

Open Jiangnanjiezi opened 2 months ago

Jiangnanjiezi commented 2 months ago

Search before asking

What happened

When I use MySQL CDC to synchronize data to Kafka, I can only get initialized data. Subsequent the data changes, whether add or delet, cannot capture the changed data. The data in Kafka is not change。

SeaTunnel Version

2.3.5

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 10000
}

source {
  MySQL-CDC {
    base-url = "jdbc:mysql://up01:3306/hive_data"
    username = "root"
    password = "123456"
    start.mode = "initial"
    server-time-zone = "Asia/Shanghai"
    exactly_once = "true"
    table-names = ["hive_data.shop_order"]
    table-names-config = [
      {
        table = "hive_data.shop_order"
        primaryKeys = ["id"]
      }
    ]

    format = compatible_debezium_json
    debezium = {
        snapshot.mode = "never"
        # include schema into kafka message
        key.converter.schemas.enable = false
        value.converter.schemas.enable = false
        # include ddl
        include.schema.changes = true
        # topic prefix
        database.server.name =  "mysql_cdc"
    }
  }
}

sink {
  kafka {
      topic = "test_topic"
      bootstrap.servers = "up01:9092"
      format = compatible_debezium_json
      kafka.request.timeout.ms = 60000
      semantics = EXACTLY_ONCE
      transaction_prefix = "sot"
      kafka.config = {
        acks = "all"
        request.timeout.ms = 60000
        buffer.memory = 33554432
      }
  }
}

Running Command

cd /export/server/apache-seatunnel-2.3.5/
./bin/seatunnel.sh --config ./config/job/shop_order_to_kafka.config -e local

Error Exception

none

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

zhilinli123 commented 1 month ago

Can you provide DDL