[Bug] [seatunnel-connectors-v2] when I use connector-cdc-mysql to read mysql-bin data,a ClassCastException error is reported

[hanyuncai]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:166) at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:82) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassCastException: io.debezium.relational.TableId cannot be cast to io.debezium.relational.TableId at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.createFetchTaskContext(MySqlDialect.java:121) at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.createFetchTaskContext(MySqlDialect.java:54) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader.checkSplitOrStartNext(IncrementalSourceSplitReader.java:132) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:64) at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.FetchTask.run(FetchTask.java:53) at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:163) ... 6 more

SeaTunnel Version

seatunnel=2.3.0

SeaTunnel Config

env{
  execution.parallelism=1
  job.mode = "STREAMING"
  job.name = "seatunnel-mysql-cdc"
  jars = "/opt/servers/seatunnel-2.3.0/connectors/seatunnel/connector-cdc-mysql-2.3.0.jar"
  checkpoint.interval = 1000
}

source {
  MySQL-CDC {
    result_table_name = "mysql-cdc"
    parallelism = 1
    server-id = 5656
    port = 3306
    hostname = "xxx.xxx.xxx.xx"
    username = "kingshardadmin"
    password = "***********"
    database-name = "test"
    table-name = "table1"
    base-url = "jdbc:mysql://xxx.xxx.xxx.xx:3306"
    startup.mode = "INITIAL"
    stop.mode = "NEVER"
#   server-time-zone = "UTC+8"
  }
}

transform{

}
sink{
   console{
   }
}

Running Command

./bin/seatunnel.sh --config ./config/example/streaming/mysql_cdc_to_console.template -e local

Error Exception

[MySQL-CDC-SourceTask (1/1)] end with state FAILED and Exception: java.lang.RuntimeException: One or more fetchers have encountered exception
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:143)
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:155)
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:91)
        at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:94)
        at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:115)
        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:68)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:157)
        at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:74)
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:357)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:166)
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:82)
        ... 5 more
Caused by: java.lang.ClassCastException: io.debezium.relational.TableId cannot be cast to io.debezium.relational.TableId
        at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.createFetchTaskContext(MySqlDialect.java:121)
        at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.createFetchTaskContext(MySqlDialect.java:54)
        at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader.checkSplitOrStartNext(IncrementalSourceSplitReader.java:132)
        at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:64)
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.FetchTask.run(FetchTask.java:53)
        at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:163)
        ... 6 more

Flink or Spark Version

flink=1.13.0 spark=2.3.0

Java or Scala Version

java=1.8 scala=2.11.12

Screenshots

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[2000liux]

please assigner to me. I try to solve this problem。

[sunnyzhuzhu]

Hi ~ This bug maybe fixed in dev branch code. I have found this problem in version 2.3.0, but this question not appears in dev code. @hanyuncai

[TyrantLucifer]

Bug is fixed, if you have any problems with this, please reopen it and discuss with community.

[simonChen0317]

2.3.0 still has this bug

[simonChen0317]

@TyrantLucifer fix in 2.3.0？

[javaonepiece]

2.3.0确实还有这个bug

[simonChen0317]

@TyrantLucifer how fix it？

[TyrantLucifer]

dev branch, 2.3.1 release.

[simonChen0317]

@TyrantLucifer noe found dev branch, 2.3.1 release.。 is 2.3.0-release branch？ has builded tar.gz？

[TyrantLucifer]

@TyrantLucifer noe found dev branch, 2.3.1 release.。 is 2.3.0-release branch？ has builded tar.gz？ 2.3.1 has not be released. You can build the dev branch by yourself.

[feng01world]

also blocked by this issue in 2.3.0 for mysql cdc. I decide to build it from source code of 2.3.1 SNAPSHOT

[feng01world]

I have realized it successfully with 2.31-SNAPTSHOT. /

some problems solved:

1. make sure mysql bin log format is ROW mode.
2. grant some super priviledges to the mysql account configed in the seatunnel config file. if not correct, you can find the notification from the seatunnel log.

[sangeetha1706]

I have realized it successfully with 2.31-SNAPTSHOT. /

some problems solved:

1. make sure mysql bin log format is ROW mode.
2. grant some super priviledges to the mysql account configed in the seatunnel config file. if not correct, you can find the notification from the seatunnel log.

when im updating the source table im not getting seatunnelRow#kind=update

[fengcheche]

2.3.3还有这个错误

[itfuyun]

我用的2.3.3版本 也一直这个错误

[pietiaonan]

我用的2.3.3版本 也一直这个错误

我也一直报这个错误，请问解决了吗

[yangwan216]

2.3.3版本中还是有这个错误，看前面的回复说2.3.1已经解决？

[simonChen0317]

我也不太清楚，不知道是否解决发自我的手机-------- 原始邮件 --------发件人： Angel Yang @.>日期： 2024年4月22日周一 15:02收件人： apache/seatunnel @.>抄送： simonChen0317 @.>, Comment @.>主 题： Re: [apache/seatunnel] [Bug] [seatunnel-connectors-v2] when I use connector-cdc-mysql to read mysql-bin data,a ClassCastException error is reported (Issue #3929) 2.3.3版本中还是有这个错误，看前面的回复说2.3.1已经解决？ 2024-04-22.15.01.42.png (view on web)

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[gmlworld]

按网上的教程部署中，有一步操作是将 connnect下的jar包拷贝到lib目录下，使用sh脚本启动的时候加载了两份相同的jar，导致了对象冲突。删除lib中拷贝过去的connect-*.jar 就没问题了。