search

apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.62k stars 1.9k forks source link

Support to run flink-cdc on Flink v1.14.0 #478

Closed macdoor closed 6 months ago

macdoor commented 2 years ago
  1. upgrade to flink-1.14.0
  2. put flink-format-changelog-json-2.0.2.jar and flink-sql-connector-mysql-cdc-2.0.2.jar into flink/lib dir.
  3. execute sql-client.sh and get exception
Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66)
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246)
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87)
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88)
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187)
    ... 1 more
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:625)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:376)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156)
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider com.ververica.cdc.formats.json.ChangelogJsonFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:621)
    ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/formats/json/JsonOptions
    at com.ververica.cdc.formats.json.ChangelogJsonFormatFactory.<clinit>(ChangelogJsonFormatFactory.java:55)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.formats.json.JsonOptions
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 21 more
leonardBang commented 2 years ago

Flink CDC 2.0. is compiled against Flink 1.13.1, this may happen if the Flink 1.13.1 is not compatible with Flink 1.14.0, it's not a bug causing be Flink CDC or Flink, what we can do is pushing the Flink's version more stable in Flink community, Or we need to offer a Flink CDC 2.0. that compiled against Flink 1.14.

macdoor commented 2 years ago

@leonardBang you are right, this is not a bug. need to offer a Flink CDC 2.0.* that compiled against Flink 1.14.

wuchong commented 2 years ago

image

voyagertanyao commented 2 years ago

we are facing the same problem, i choose flink-cdc 2.0.2 and flink 1.14, not successful. changing flink to 1.13.1 just works

yandavid163 commented 2 years ago

的确不行，flink-sql-connector-mysql-cdc-2.1.0 放进来就报错。

Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue. at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201) at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) Caused by: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165) at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) at org.apache.flink.table.client.gateway.context.ExecutionContext.(ExecutionContext.java:66) at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) ... 1 more Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories. at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:625) at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:376) at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ... 8 more Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider com.ververica.cdc.formats.json.ChangelogJsonFormatFactory could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at java.util.Iterator.forEachRemaining(Iterator.java:116) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:621) ... 10 more Caused by: java.lang.NoClassDefFoundError: org/apache/flink/formats/json/JsonOptions at com.ververica.cdc.formats.json.ChangelogJsonFormatFactory.(ChangelogJsonFormatFactory.java:55) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 14 more Caused by: java.lang.ClassNotFoundException: org.apache.flink.formats.json.JsonOptions

macdoor commented 2 years ago

flink 1.14 does not compatible with 1.13

wangpengfei2048 commented 2 years ago

How is the problem? I am facing the same problem, too. I use flink-sql-connector-mysql-cdc-2.2.0.jar, but the problem exsits, too. 1, I use flink-1.14.3 2, put flink-format-changelog-json-2.0.2.jar and flink-sql-connector-mysql-cdc-2.2.0.jar into flink/lib dir. 3, execute sql-client.sh and get exception flink-sql-connector-mysql-cdc-2.2.0.jar is mentioned compiled with 1.14, but the problem exsits. I think the problem is in flink-format-changelog-json-2.0.2.jar , when I only put flink-format-changelog-json-2.0.2.jar into flink/lib dir, and execute sql-client.sh, the problem happens. But I dont see any update with the problem https://github.com/ververica/flink-cdc-connectors/pull/756

PatrickRen commented 6 months ago

Closing this issue because it was created before version 2.3.0 (2022-11-10). Please try the latest version of Flink CDC to see if the issue has been resolved. If the issue is still valid, kindly report it on Apache Jira under project Flink with component tag Flink CDC. Thank you!