apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.63k stars 1.91k forks source link

[flink][flink-cdc] version incompatible #1768

Closed Capri0110 closed 6 months ago

Capri0110 commented 1 year ago

Steps to reproduce the behavior:

  1. The test data :
  2. The test code :

    
    public class FlinkCdcStreamTest {
    
    public static void main(String[] args) throws Exception {
        MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
                .hostname("localhost")
                .port(3306)
                // support fully regexp content
                .databaseList("test_db") // set captured database
                .tableList("^test(.*).t_cdc_bench_[0-9]$") // set captured table(s), in context of args ...
                .username("root")
                .password("root")
                // converts SourceRecord to JSON String
                .deserializer(new JsonDebeziumDeserializationSchema(false))
                .startupOptions(StartupOptions.latest())
                .includeSchemaChanges(true)
                .serverTimeZone("GMT+7")
                .serverId("5300")
                .build();
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStream<String> streamSource = env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "my-source")
                .setParallelism(1);
    
        streamSource.print();
    
        env.execute();
    }

}

4. The error :
flink version: 1.14.3; flink-cdc version: 2.3.0 

十一月 22, 2022 7:09:17 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect 信息: Connected to localhost:3306 at binlog.000036/447702 (sid:5300, cid:651) 3> {"before":{"order_id":1020,"order_date":19290,"order_time":1666732712000,"quantity":1009,"product_id":110,"purchaser":"pubg","country":"us","ccr_code":"FA==","mysql_ver":"UA==","acct":null},"after":{"order_id":1020,"order_date":19290,"order_time":1666732712000,"quantity":100,"product_id":110,"purchaser":"pubg","country":"us","ccr_code":"FA==","mysql_ver":"UA==","acct":null},"source":{"version":"1.6.4.Final","connector":"mysql","name":"mysql_binlog_source","ts_ms":1669115382000,"snapshot":"false","db":"test_db","sequence":null,"table":"t_cdc_bench_1","server_id":1,"gtid":null,"file":"binlog.000036","pos":447952,"row":0,"thread":null,"query":null},"op":"u","ts_ms":1669115382756,"transaction":null}

console print sink fine

flink version: 1.13.3; flink-cdc version: 2.2.1 

十一月 22, 2022 7:16:02 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect 信息: Connected to localhost:3306 at binlog.000036/448115 (sid:5300, cid:656) 5> {"before":{"order_id":1020,"order_date":19290,"order_time":1666732712000,"quantity":100,"product_id":110,"purchaser":"pubg","country":"us","ccr_code":"FA==","mysql_ver":"UA==","acct":null},"after":{"order_id":1020,"order_date":19290,"order_time":1666732712000,"quantity":1009,"product_id":110,"purchaser":"pubg","country":"us","ccr_code":"FA==","mysql_ver":"UA==","acct":null},"source":{"version":"1.5.4.Final","connector":"mysql","name":"mysql_binlog_source","ts_ms":1669115769000,"snapshot":"false","db":"test_db","sequence":null,"table":"t_cdc_bench_1","server_id":1,"gtid":null,"file":"binlog.000036","pos":448365,"row":0,"thread":null,"query":null},"op":"u","ts_ms":1669115769583,"transaction":null}

console print sink also fine

flink version: 1.13.3; flink-cdc version: 2.3.0 

/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/bin/java -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=63084:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/charsets.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/cldrdata.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/dnsns.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/jaccess.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/jfxrt.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/localedata.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/nashorn.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/sunec.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/ext/zipfs.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/jce.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/jfr.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/jfxswt.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/jsse.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/management-agent.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/resources.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/jre/lib/rt.jar:/Users/nirvana.xu/IdeaProjects/local/flink_test/target/classes:/Users/nirvana.xu/.m2/repository/org/apache/logging/log4j/log4j-core/2.17.1/log4j-core-2.17.1.jar:/Users/nirvana.xu/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar:/Users/nirvana.xu/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.1/log4j-api-2.17.1.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-clients_2.11/1.13.3/flink-clients_2.11-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-runtime_2.11/1.13.3/flink-runtime_2.11-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-queryable-state-client-java/1.13.3/flink-queryable-state-client-java-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-hadoop-fs/1.13.3/flink-hadoop-fs-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-shaded-netty/4.1.49.Final-13.0/flink-shaded-netty-4.1.49.Final-13.0.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-shaded-jackson/2.12.1-13.0/flink-shaded-jackson-2.12.1-13.0.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-shaded-zookeeper-3/3.4.14-13.0/flink-shaded-zookeeper-3-3.4.14-13.0.jar:/Users/nirvana.xu/.m2/repository/org/javassist/javassist/3.24.0-GA/javassist-3.24.0-GA.jar:/Users/nirvana.xu/.m2/repository/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/Users/nirvana.xu/.m2/repository/com/typesafe/akka/akka-actor_2.11/2.5.21/akka-actor_2.11-2.5.21.jar:/Users/nirvana.xu/.m2/repository/com/typesafe/config/1.3.3/config-1.3.3.jar:/Users/nirvana.xu/.m2/repository/org/scala-lang/modules/scala-java8-compat_2.11/0.7.0/scala-java8-compat_2.11-0.7.0.jar:/Users/nirvana.xu/.m2/repository/com/typesafe/akka/akka-stream_2.11/2.5.21/akka-stream_2.11-2.5.21.jar:/Users/nirvana.xu/.m2/repository/org/reactivestreams/reactive-streams/1.0.2/reactive-streams-1.0.2.jar:/Users/nirvana.xu/.m2/repository/com/typesafe/ssl-config-core_2.11/0.3.7/ssl-config-core_2.11-0.3.7.jar:/Users/nirvana.xu/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.1.1/scala-parser-combinators_2.11-1.1.1.jar:/Users/nirvana.xu/.m2/repository/com/typesafe/akka/akka-protobuf_2.11/2.5.21/akka-protobuf_2.11-2.5.21.jar:/Users/nirvana.xu/.m2/repository/com/typesafe/akka/akka-slf4j_2.11/2.5.21/akka-slf4j_2.11-2.5.21.jar:/Users/nirvana.xu/.m2/repository/org/clapper/grizzled-slf4j_2.11/1.3.2/grizzled-slf4j_2.11-1.3.2.jar:/Users/nirvana.xu/.m2/repository/com/github/scopt/scopt_2.11/3.5.0/scopt_2.11-3.5.0.jar:/Users/nirvana.xu/.m2/repository/org/xerial/snappy/snappy-java/1.1.8.3/snappy-java-1.1.8.3.jar:/Users/nirvana.xu/.m2/repository/com/twitter/chill_2.11/0.7.6/chill_2.11-0.7.6.jar:/Users/nirvana.xu/.m2/repository/com/twitter/chill-java/0.7.6/chill-java-0.7.6.jar:/Users/nirvana.xu/.m2/repository/org/lz4/lz4-java/1.6.0/lz4-java-1.6.0.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-optimizer_2.11/1.13.3/flink-optimizer_2.11-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-java/1.13.3/flink-java-1.13.3.jar:/Users/nirvana.xu/.m2/repository/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar:/Users/nirvana.xu/.m2/repository/org/slf4j/slf4j-api/1.7.15/slf4j-api-1.7.15.jar:/Users/nirvana.xu/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/force-shading/1.13.3/force-shading-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-table-uber-blink_2.11/1.13.3/flink-table-uber-blink_2.11-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-sql-parser/1.13.3/flink-sql-parser-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-sql-parser-hive/1.13.3/flink-sql-parser-hive-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-core/1.13.3/flink-core-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-annotations/1.13.3/flink-annotations-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-metrics-core/1.13.3/flink-metrics-core-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-shaded-asm-7/7.1-13.0/flink-shaded-asm-7-7.1-13.0.jar:/Users/nirvana.xu/.m2/repository/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar:/Users/nirvana.xu/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/Users/nirvana.xu/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/Users/nirvana.xu/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-shaded-guava/18.0-13.0/flink-shaded-guava-18.0-13.0.jar:/Users/nirvana.xu/.m2/repository/com/ververica/flink-connector-mysql-cdc/2.3.0/flink-connector-mysql-cdc-2.3.0.jar:/Users/nirvana.xu/.m2/repository/com/ververica/flink-connector-debezium/2.3.0/flink-connector-debezium-2.3.0.jar:/Users/nirvana.xu/.m2/repository/io/debezium/debezium-api/1.6.4.Final/debezium-api-1.6.4.Final.jar:/Users/nirvana.xu/.m2/repository/io/debezium/debezium-embedded/1.6.4.Final/debezium-embedded-1.6.4.Final.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/connect-api/2.7.1/connect-api-2.7.1.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/kafka-clients/2.7.1/kafka-clients-2.7.1.jar:/Users/nirvana.xu/.m2/repository/com/github/luben/zstd-jni/1.4.5-6/zstd-jni-1.4.5-6.jar:/Users/nirvana.xu/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/connect-runtime/2.7.1/connect-runtime-2.7.1.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/kafka-tools/2.7.1/kafka-tools-2.7.1.jar:/Users/nirvana.xu/.m2/repository/net/sourceforge/argparse4j/argparse4j/0.7.0/argparse4j-0.7.0.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/connect-transforms/2.7.1/connect-transforms-2.7.1.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.10.5/jackson-jaxrs-json-provider-2.10.5.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.10.5/jackson-jaxrs-base-2.10.5.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.10.5/jackson-module-jaxb-annotations-2.10.5.jar:/Users/nirvana.xu/.m2/repository/jakarta/xml/bind/jakarta.xml.bind-api/2.3.2/jakarta.xml.bind-api-2.3.2.jar:/Users/nirvana.xu/.m2/repository/jakarta/activation/jakarta.activation-api/1.2.1/jakarta.activation-api-1.2.1.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.31/jersey-container-servlet-2.31.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.31/jersey-container-servlet-core-2.31.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/hk2/external/jakarta.inject/2.6.1/jakarta.inject-2.6.1.jar:/Users/nirvana.xu/.m2/repository/jakarta/ws/rs/jakarta.ws.rs-api/2.1.6/jakarta.ws.rs-api-2.1.6.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/jersey/inject/jersey-hk2/2.31/jersey-hk2-2.31.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/hk2/hk2-locator/2.6.1/hk2-locator-2.6.1.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.6.1/aopalliance-repackaged-2.6.1.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/hk2/hk2-api/2.6.1/hk2-api-2.6.1.jar:/Users/nirvana.xu/.m2/repository/org/glassfish/hk2/hk2-utils/2.6.1/hk2-utils-2.6.1.jar:/Users/nirvana.xu/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-servlets/9.4.38.v20210224/jetty-servlets-9.4.38.v20210224.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-continuation/9.4.38.v20210224/jetty-continuation-9.4.38.v20210224.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-client/9.4.38.v20210224/jetty-client-9.4.38.v20210224.jar:/Users/nirvana.xu/.m2/repository/org/reflections/reflections/0.9.12/reflections-0.9.12.jar:/Users/nirvana.xu/.m2/repository/org/apache/maven/maven-artifact/3.6.3/maven-artifact-3.6.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/connect-json/2.7.1/connect-json-2.7.1.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.10.5/jackson-datatype-jdk8-2.10.5.jar:/Users/nirvana.xu/.m2/repository/org/apache/kafka/connect-file/2.7.1/connect-file-2.7.1.jar:/Users/nirvana.xu/.m2/repository/io/debezium/debezium-connector-mysql/1.6.4.Final/debezium-connector-mysql-1.6.4.Final.jar:/Users/nirvana.xu/.m2/repository/io/debezium/debezium-core/1.6.4.Final/debezium-core-1.6.4.Final.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.10.5/jackson-datatype-jsr310-2.10.5.jar:/Users/nirvana.xu/.m2/repository/io/debezium/debezium-ddl-parser/1.6.4.Final/debezium-ddl-parser-1.6.4.Final.jar:/Users/nirvana.xu/.m2/repository/org/antlr/antlr4-runtime/4.8/antlr4-runtime-4.8.jar:/Users/nirvana.xu/.m2/repository/com/zendesk/mysql-binlog-connector-java/0.25.3/mysql-binlog-connector-java-0.25.3.jar:/Users/nirvana.xu/.m2/repository/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar:/Users/nirvana.xu/.m2/repository/com/esri/geometry/esri-geometry-api/2.2.0/esri-geometry-api-2.2.0.jar:/Users/nirvana.xu/.m2/repository/com/zaxxer/HikariCP/4.0.3/HikariCP-4.0.3.jar:/Users/nirvana.xu/.m2/repository/org/awaitility/awaitility/4.0.1/awaitility-4.0.1.jar:/Users/nirvana.xu/.m2/repository/org/hamcrest/hamcrest/2.1/hamcrest-2.1.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-streaming-java_2.11/1.13.3/flink-streaming-java_2.11-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/flink/flink-file-sink-common/1.13.3/flink-file-sink-common-1.13.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/commons/commons-math3/3.5/commons-math3-3.5.jar:/Users/nirvana.xu/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/nirvana.xu/.m2/repository/commons-configuration/commons-configuration/1.10/commons-configuration-1.10.jar:/Users/nirvana.xu/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/nirvana.xu/Library/Java/JavaVirtualMachines/corretto-1.8.0_312/Contents/Home/lib/tools.jar:/Users/nirvana.xu/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar:/Users/nirvana.xu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/Users/nirvana.xu/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-server/9.3.19.v20170502/jetty-server-9.3.19.v20170502.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-http/9.3.19.v20170502/jetty-http-9.3.19.v20170502.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-io/9.3.19.v20170502/jetty-io-9.3.19.v20170502.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-util/9.3.19.v20170502/jetty-util-9.3.19.v20170502.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-servlet/9.3.19.v20170502/jetty-servlet-9.3.19.v20170502.jar:/Users/nirvana.xu/.m2/repository/org/eclipse/jetty/jetty-security/9.3.19.v20170502/jetty-security-9.3.19.v20170502.jar:/Users/nirvana.xu/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/nirvana.xu/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.7.8/jackson-databind-2.7.8.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.7.8/jackson-core-2.7.8.jar:/Users/nirvana.xu/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar:/Users/nirvana.xu/.m2/repository/org/apache/commons/commons-lang3/3.11/commons-lang3-3.11.jar:/Users/nirvana.xu/.m2/repository/org/apache/poi/poi/4.1.2/poi-4.1.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/Users/nirvana.xu/.m2/repository/com/zaxxer/SparseBitSet/1.2/SparseBitSet-1.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/poi/poi-ooxml/4.1.2/poi-ooxml-4.1.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/poi/poi-ooxml-schemas/4.1.2/poi-ooxml-schemas-4.1.2.jar:/Users/nirvana.xu/.m2/repository/org/apache/xmlbeans/xmlbeans/3.1.0/xmlbeans-3.1.0.jar:/Users/nirvana.xu/.m2/repository/com/github/virtuald/curvesapi/1.06/curvesapi-1.06.jar:/Users/nirvana.xu/.m2/repository/com/google/code/gson/gson/2.8.5/gson-2.8.5.jar:/Users/nirvana.xu/.m2/repository/com/amazonaws/aws-java-sdk-s3/1.11.271/aws-java-sdk-s3-1.11.271.jar:/Users/nirvana.xu/.m2/repository/com/amazonaws/aws-java-sdk-kms/1.11.271/aws-java-sdk-kms-1.11.271.jar:/Users/nirvana.xu/.m2/repository/com/amazonaws/aws-java-sdk-core/1.11.271/aws-java-sdk-core-1.11.271.jar:/Users/nirvana.xu/.m2/repository/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar:/Users/nirvana.xu/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar:/Users/nirvana.xu/.m2/repository/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar:/Users/nirvana.xu/.m2/repository/com/amazonaws/jmespath-java/1.11.271/jmespath-java-1.11.271.jar:/Users/nirvana.xu/.m2/repository/com/amazonaws/aws-java-sdk-cognitoidp/1.11.271/aws-java-sdk-cognitoidp-1.11.271.jar:/Users/nirvana.xu/.m2/repository/com/alibaba/fastjson/1.2.58/fastjson-1.2.58.jar:/Users/nirvana.xu/.m2/repository/org/projectlombok/lombok/1.18.24/lombok-1.18.24.jar FlinkCdcStreamTest 19:17:40.953 [Source Data Fetcher for Source: my-source (1/1)#0] ERROR org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager - Received uncaught exception. java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava30/com/google/common/util/concurrent/ThreadFactoryBuilder at com.ververica.cdc.connectors.mysql.debezium.reader.BinlogSplitReader.(BinlogSplitReader.java:93) ~[flink-connector-mysql-cdc-2.3.0.jar:2.3.0] at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.checkSplitOrStartNext(MySqlSplitReader.java:156) ~[flink-connector-mysql-cdc-2.3.0.jar:2.3.0] at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.fetch(MySqlSplitReader.java:71) ~[flink-connector-mysql-cdc-2.3.0.jar:2.3.0] at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56) ~[flink-table-uber-blink_2.11-1.13.3.jar:1.13.3] at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:140) ~[flink-table-uber-blink_2.11-1.13.3.jar:1.13.3] at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:103) [flink-table-uber-blink_2.11-1.13.3.jar:1.13.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_312] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312] Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_312] at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_312] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_312] at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_312] ... 11 more



According to the official doc, the cdc 2.3.0 should support flink 1.13.x
<img width="405" alt="截屏2022-11-22 下午7 18 54" src="https://user-images.githubusercontent.com/44332181/203301370-e700d085-0154-490a-abf5-a874449ab302.png">
Further, as mentioned in https://github.com/ververica/flink-cdc-connectors/issues/1748, I think there are conflicts with cdc 2.3.0 and flink-core 1.13.x

Looking forward to feedbacks, as we want to try cdc 2.3.0 with specific-offsets feature implemented :)
tuoluzhe8521 commented 1 year ago

when i use flink version: 1.13.0, it also have this problem

Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava30/com/google/common/util/concurrent/ThreadFactoryBuilder at com.ververica.cdc.connectors.mysql.debezium.reader.BinlogSplitReader.(BinlogSplitReader.java:93) at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.checkSplitOrStartNext(MySqlSplitReader.java:156) at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.fetch(MySqlSplitReader.java:71) at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:56) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:138) at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:101) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more

dorajamie commented 1 year ago

https://github.com/ververica/flink-cdc-connectors/issues/1845

PatrickRen commented 6 months ago

Duplicated #1845