search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.06k stars 1.83k forks source link

MYSQL CDC到Paimon0.9 flink1.9版本，提交任务会报：Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype #8049

Open lxxstarii opened 1 week ago

lxxstarii commented 1 week ago

Search before asking

What happened

MYSQL CDC到Paimon0.9 flink1.9版本，提交任务会报：Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype

SeaTunnel Version

2.3.8

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  Mysql-CDC {
    base-url = "jdbc:mysql://192.168.0.44:3306/company"
    username = "root"
    password = "000000"
    table-names = ["company.userinfo2"]
  }
}

transform {
}

sink {
  Paimon {
    catalog_name="paimon_hive"
    warehouse="hdfs://hadoop100:8020/paimon/hive"
    database="default"
    table="test_paimon_03"
  }
}

Running Command

start-seatunnel-flink-15-connector-v2.sh

Error Exception

file:/root/Downloads/home/software/apache-seatunnel-2.3.8/starter/seatunnel-flink-15-starter.jar
2024-11-13 19:25:07,517 INFO  org.apache.seatunnel.core.starter.flink.execution.FlinkExecution [] - Inject jar to config: file:/root/Downloads/home/software/apache-seatunnel-2.3.8/lib/seatunnel-hadoop3-3.1.4-uber.jar
2024-11-13 19:25:07,517 INFO  org.apache.seatunnel.core.starter.flink.execution.FlinkExecution [] - Inject jar to config: file:/root/Downloads/home/software/apache-seatunnel-2.3.8/connectors/connector-cdc-mysql-2.3.8.jar
2024-11-13 19:25:07,518 INFO  org.apache.seatunnel.core.starter.flink.execution.FlinkExecution [] - Inject jar to config: file:/root/Downloads/home/software/apache-seatunnel-2.3.8/connectors/connector-paimon-2.3.8.jar
2024-11-13 19:25:07,644 INFO  org.apache.flink.api.java.typeutils.TypeExtractor            [] - class org.apache.seatunnel.api.table.type.SeaTunnelRow does not contain a setter for field fields
2024-11-13 19:25:07,644 INFO  org.apache.flink.api.java.typeutils.TypeExtractor            [] - Class class org.apache.seatunnel.api.table.type.SeaTunnelRow cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution.
2024-11-13 19:25:07,667 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery [] - Load SeaTunnelSink Plugin from /root/Downloads/home/software/apache-seatunnel-2.3.8/connectors
2024-11-13 19:25:07,710 INFO  org.apache.seatunnel.connectors.seatunnel.paimon.security.PaimonSecurityContext [] - Hadoop config initialized: org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonHadoopConfiguration
2024-11-13 19:25:07,731 INFO  org.apache.seatunnel.connectors.seatunnel.paimon.security.PaimonSecurityContext [] - Hadoop config initialized: org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonHadoopConfiguration
2024-11-13 19:25:07,758 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - 

===============================================================================

2024-11-13 19:25:07,758 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Fatal Error, 

2024-11-13 19:25:07,759 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-11-13 19:25:07,759 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Reason:Flink job executed failed 

2024-11-13 19:25:07,761 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: Flink job executed failed
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:63)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
    at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
    at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
    at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:217)
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143)
    at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115)
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61)
    ... 19 more
Caused by: org.apache.seatunnel.connectors.seatunnel.paimon.exception.PaimonConnectorException: ErrorCode:[PAIMON-06], ErrorDescription:[Load catalog failed] - Failed to perform SecurityContext.runSecured
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:89)
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73)
    at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64)
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214)
    ... 22 more
Caused by: java.lang.RuntimeException: Unexpected error when trying to load service provider.
    at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:118)
    at org.apache.paimon.factories.FactoryUtil.discoverFactory(FactoryUtil.java:38)
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:73)
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66)
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.lambda$loadCatalog$1(PaimonCatalogLoader.java:87)
    at org.apache.paimon.security.SecurityContext.runSecured(SecurityContext.java:67)
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:86)
    ... 25 more
Caused by: java.util.ServiceConfigurationError: org.apache.paimon.factories.Factory: Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype
    at java.util.ServiceLoader.fail(ServiceLoader.java:239)
    at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:109)
    ... 31 more

2024-11-13 19:25:07,761 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - 
===============================================================================

2024-11-13 19:25:07,762 ERROR org.apache.flink.client.cli.CliFrontend                      [] - Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Flink job executed failed
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108) ~[flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) ~[flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) ~[flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) ~[flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) ~[flink-dist-1.19.1.jar:1.19.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
    at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) [flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar:3.1.1.7.1.1.0-565-9.0]
    at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) [flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) [flink-dist-1.19.1.jar:1.19.1]
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) [flink-dist-1.19.1.jar:1.19.1]
Caused by: org.apache.seatunnel.core.starter.exception.CommandExecuteException: Flink job executed failed
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:63) ~[?:?]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
    ... 12 more
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:217) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
    ... 12 more
Caused by: org.apache.seatunnel.connectors.seatunnel.paimon.exception.PaimonConnectorException: ErrorCode:[PAIMON-06], ErrorDescription:[Load catalog failed] - Failed to perform SecurityContext.runSecured
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:89) ~[?:?]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73) ~[?:?]
    at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
    ... 12 more
Caused by: java.lang.RuntimeException: Unexpected error when trying to load service provider.
    at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:118) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.paimon.factories.FactoryUtil.discoverFactory(FactoryUtil.java:38) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:73) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.lambda$loadCatalog$1(PaimonCatalogLoader.java:87) ~[?:?]
    at org.apache.paimon.security.SecurityContext.runSecured(SecurityContext.java:67) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:86) ~[?:?]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73) ~[?:?]
    at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
    ... 12 more
Caused by: java.util.ServiceConfigurationError: org.apache.paimon.factories.Factory: Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype
    at java.util.ServiceLoader.fail(ServiceLoader.java:239) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$300(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:109) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.paimon.factories.FactoryUtil.discoverFactory(FactoryUtil.java:38) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:73) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.lambda$loadCatalog$1(PaimonCatalogLoader.java:87) ~[?:?]
    at org.apache.paimon.security.SecurityContext.runSecured(SecurityContext.java:67) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:86) ~[?:?]
    at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73) ~[?:?]
    at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
    at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
    ... 12 more

Zeta or Flink or Spark Version

flink 1.9 Paimon0.9

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct