search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.08k stars 1.83k forks source link

MySQL-CDC SparkTaskExecuteCommand: Run SeaTunnel on spark failed. #4948

Closed yunfeng79 closed 1 year ago

yunfeng79 commented 1 year ago

Search before asking

What happened

apache-seatunnel-incubating-2.3.1 spark-2.4.5

env {

You can set SeaTunnel environment configuration here

execution.parallelism = 2 job.mode = "BATCH" checkpoint.interval = 10000

execution.checkpoint.interval = 10000

execution.checkpoint.data-uri = "hdfs://dc-01:8020/flink-checkpoints"

}

source {

MySQL-CDC { base-url = "jdbc:mysql://192.168.1.39:3306/testdb" parallelism = 1 server-id = 7 user = "dev" password = "dev" query = "select id,name,age from user1" table-name = ["testdb.user1"] result_table_name = "res011" }

}

sink { Doris { fenodes = "192.168.1.231:8030" username = dev password = "dev" table.identifier = "test.user2" sink.enable-2pc = "true" sink.label-prefix = "user2_1" doris.config = { format = "csv" column_separator = "," } } }

image

SeaTunnel Version

apache-seatunnel-incubating-2.3.1

SeaTunnel Config

env {
  # You can set SeaTunnel environment configuration here
  execution.parallelism = 2
  job.mode = "BATCH"
  checkpoint.interval = 10000
  #execution.checkpoint.interval = 10000
  #execution.checkpoint.data-uri = "hdfs://dc-01:8020/flink-checkpoints"
}

source {

  MySQL-CDC { 
    base-url = "jdbc:mysql://192.168.1.39:3306/testdb"
    parallelism = 1 
    server-id = 7
    user = "dev"
    password = "dev"
    query = "select id,`name`,age from user1"
    table-name = ["testdb.user1"]
    result_table_name = "res011"
   }

}

sink {
  Doris {
    fenodes = "192.168.1.231:8030"
    username = dev
    password = "dev"
    table.identifier = "test.user2"
    sink.enable-2pc = "true"
    sink.label-prefix = "user2_1"
    doris.config = {
          format = "csv"
          column_separator = ","
      }
   }
}

Running Command

./bin/start-seatunnel-spark-2-connector-v2.sh --master local[4] --deploy-mode client --config  ./config/v2-cdc-mysqltodoris.conf

Error Exception

23/06/19 15:23:48 INFO AbstractPluginDiscovery: Load SeaTunnelSource Plugin from /home/app/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
23/06/19 15:23:48 INFO AbstractPluginDiscovery: Discovery plugin jar: MySQL-CDC at: file:/home/app/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-cdc-mysql-2.3.1.jar
23/06/19 15:23:48 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='MySQL-CDC'} from classpath
23/06/19 15:23:48 ERROR SparkTaskExecuteCommand: Run SeaTunnel on spark failed.
java.lang.NullPointerException
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
        at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.config.MySqlSourceConfigFactory.create(MySqlSourceConfigFactory.java:61)
        at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlDialect.<init>(MySqlDialect.java:53)
        at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.MySqlIncrementalSource.createDataSourceDialect(MySqlIncrementalSource.java:128)
        at org.apache.seatunnel.connectors.cdc.base.source.IncrementalSource.prepare(IncrementalSource.java:117)
        at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:104)
        at org.apache.seatunnel.core.starter.spark.execution.SparkAbstractPluginExecuteProcessor.<init>(SparkAbstractPluginExecuteProcessor.java:49)
        at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:51)
        at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:57)
        at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:59)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/06/19 15:23:48 ERROR SeaTunnel: 

===============================================================================

23/06/19 15:23:48 ERROR SeaTunnel: Fatal Error, 

23/06/19 15:23:48 ERROR SeaTunnel: Please submit bug report in https://github.com/apache/incubator-seatunnel/issues

23/06/19 15:23:48 ERROR SeaTunnel: Reason:null 

23/06/19 15:23:48 ERROR SeaTunnel: Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException
        at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:63)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

23/06/19 15:23:48 ERROR SeaTunnel:

Flink or Spark Version

spark 2.4.5

Java or Scala Version

1.8.0_261

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 year ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.