apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.06k stars 1.83k forks source link

[Bug] [JDBC-HIVE] hive as source, #8032

Closed NoPr closed 1 week ago

NoPr commented 1 week ago

Search before asking

What happened

Using jdbc -hive as source, executing the script reports an error。The specified class cannot be found, but the jar package has been put in. 使用hive的jdbc做为source,报错找不到指定类,但是已经放入了jar包

SeaTunnel Version

2.3.7

SeaTunnel Config

env {
    parallelism = 1
    job.mode =  BATCH
}
source {
    Jdbc {
        result_table_name =  "ta_1856172468898095104_ble"
        driver = "org.apache.hive.jdbc.HiveDriver"
        url = "jdbc:hive2://127.0.0.1:10000/userdb"
        query = "select * from 'userdb'.'t_resource_unstructured_file'"
        user = "root"
        password = ""
    }
}
transform {}
sink {
    Hive {
        source_table_name = "ta_1856172468898095104_ble"
        table_name =  "btrc.test2233"
        metastore_uri = "thrift://127.0.0.1:9083"
    }
}

Running Command

./bin/seatunnel.sh -c ***.config -e local

Error Exception

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:211)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Jdbc'.
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:101)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:375)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:209)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:114)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:182)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:158)
        ... 2 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSocketTransport(Ljava/lang/String;II)Lorg/apache/hive/org/apache/thrift/transport/TTransport;
        at org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:561)
        at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:581)
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
        at org.apache.seatunnel.connec

Zeta or Flink or Spark Version

seatunnel-2.3.7

Java or Scala Version

java1.8

Screenshots

2e70c8ad2c2dff703f48e2eb456892c image image

Are you willing to submit PR?

Code of Conduct

liangjiaxing2022 commented 1 week ago

hive jdbc插件不支持sink

NoPr commented 1 week ago

hive jdbc插件不支持sink 使用的hive是source部分的jdbc插件。sink部分使用的是metadata uri

NoPr commented 1 week ago

已解决。

xinshuaiyong commented 32 minutes ago

请问你如何解决的,我也碰到这个问题

NoPr commented 27 minutes ago

请问你如何解决的,我也碰到这个问题

我的JDK是17,我重新编译了项目,并且添加了hive-jdbc-3.1.3.jar。大小为123KB那个

xinshuaiyong commented 8 minutes ago

我这边是jdk1.8,集群,也添加了这个jar,并打包的时候会打到lib下,重新编译后,还是找不到驱动