apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.86k stars 1.77k forks source link

Exception in thread "main" java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Hudi'} not found #4398

Open zhangzhaohuazai opened 1 year ago

zhangzhaohuazai commented 1 year ago

Search before asking

What happened

I write a job to sink hudi,but I get this error: Exception in thread "main" java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Hudi'} not found

SeaTunnel Version

2.3.0

SeaTunnel Config

source {
  MySQL-CDC {
    result_table_name = "mysql_users2"
    parallelism = 1
    server-id = 5656
    port = 3306
    hostname = "***"
    username = "***"
    password = "***"
    database-name = "test1"
    table-name = "users_cdc2"
    base-url = "jdbc:mysql://***:3306"
    field_name = "id,name,birthday,ts"
  }
}

transform {
  sql{
    source_table_name = "mysql_users2"
    sql = "select *,DATE_FORMAT(birthday,'yyyyMMdd') as `partition` from mysql_users2"
    result_table_name = "mycdc_v"
  }

}

sink {
  Hudi {
    source_table_name = "mycdc_v"
    table.path = "hdfs://***:9000/seatunnel/mysql_cdc_sync_hudi_02"
    table.type = "cow"
    conf.files = "/xxx/hadoop/etc/hadoop/hdfs-site.xml;/xxx/hadoop/etc/hadoop/core-site.xml;/usr/local/hadoop/etc/hadoop/yarn-site.xml"
  }

  Hive {
    source_table_name = "mycdc_v"
    table_name = "users_cdc2.mysql_cdc_sync_hive_02"
    metastore_uri = "thrift://***:9083"
    partition_by = ["partition"]
    sink_columns = ["id", "name", "birthday", "ts","partition"]
  }
}

Running Command

./bin/seatunnel.sh --config ./config/mysqlcdc_hudi-hdfs_hive2.conf.template -e local

Error Exception

`2023-03-22 17:57:43,984 ERROR org.apache.seatunnel.core.starter.Seatunnel - Fatal Error, 

2023-03-22 17:57:43,984 ERROR org.apache.seatunnel.core.starter.Seatunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues

2023-03-22 17:57:43,984 ERROR org.apache.seatunnel.core.starter.Seatunnel - Reason:Plugin PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Hudi'} not found. 

2023-03-22 17:57:43,985 ERROR org.apache.seatunnel.core.starter.Seatunnel - Exception StackTrace:java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Hudi'} not found.
    at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:200)
    at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSinkInstance(ConnectorInstanceLoader.java:79)
    at org.apache.seatunnel.engine.core.parse.JobConfigParser.complexAnalyze(JobConfigParser.java:178)
    at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:127)
    at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:129)
    at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:121)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:91)
    at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:31)
`

Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

liugddx commented 1 year ago
hudi-hdfs_hive2.

hudi sink is not yet supported.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.