search

DTStack / chunjun

A data integration framework
https://dtstack.github.io/chunjun/
Apache License 2.0
3.98k stars 1.69k forks source link

[Bug] [chunjun-connecter-hive]org.apache.thrift.TException: Unsupported Hive2 protocol #1354

Open JIrui123 opened 1 year ago

JIrui123 commented 1 year ago

Search before asking

What happened

Caused by: com.dtstack.chunjun.throwable.ChunJunRuntimeException: connection info ：jdbc:hive2://192.168.31.56:10000/flinkx error message ：java.sql.SQLException: Could not establish connection to jdbc:hive2://192.168.31.56:10000/: Unsupported Hive2 protocol at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:601) at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:172) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.getHiveConnection(HiveDbUtil.java:257) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.connect(HiveDbUtil.java:211) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.connect(HiveDbUtil.java:203) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil$1.call(HiveDbUtil.java:96) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil$1.call(HiveDbUtil.java:93) at com.dtstack.chunjun.util.RetryUtil$Retry.call(RetryUtil.java:141) at com.dtstack.chunjun.util.RetryUtil$Retry.doRetry(RetryUtil.java:80) at com.dtstack.chunjun.util.RetryUtil.executeWithRetry(RetryUtil.java:56) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.getConnectionWithRetry(HiveDbUtil.java:92) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.getConnection(HiveDbUtil.java:86) at com.dtstack.chunjun.connector.hive.util.HiveUtil.createHiveTableWithTableInfo(HiveUtil.java:90) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.checkCreateTable(HiveOutputFormat.java:386) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.primaryCreateTable(HiveOutputFormat.java:357) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.openInternal(HiveOutputFormat.java:114) at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.open(BaseRichOutputFormat.java:262) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.open(HiveOutputFormat.java:94) at com.dtstack.chunjun.sink.DtOutputFormatSinkFunction.open(DtOutputFormatSinkFunction.java:95) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:46) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:433) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:545) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93) at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:535) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.thrift.TException: Unsupported Hive2 protocol at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:595) ... 32 more

    at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.connect(HiveDbUtil.java:222)
    at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.connect(HiveDbUtil.java:203)
    at com.dtstack.chunjun.connector.hive.util.HiveDbUtil$1.call(HiveDbUtil.java:96)
    at com.dtstack.chunjun.connector.hive.util.HiveDbUtil$1.call(HiveDbUtil.java:93)
    at com.dtstack.chunjun.util.RetryUtil$Retry.call(RetryUtil.java:141)
    at com.dtstack.chunjun.util.RetryUtil$Retry.doRetry(RetryUtil.java:80)
    ... 21 more

. at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.getConnectionWithRetry(HiveDbUtil.java:104) at com.dtstack.chunjun.connector.hive.util.HiveDbUtil.getConnection(HiveDbUtil.java:86) at com.dtstack.chunjun.connector.hive.util.HiveUtil.createHiveTableWithTableInfo(HiveUtil.java:90) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.checkCreateTable(HiveOutputFormat.java:386) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.primaryCreateTable(HiveOutputFormat.java:357) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.openInternal(HiveOutputFormat.java:114) at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.open(BaseRichOutputFormat.java:262) at com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat.open(HiveOutputFormat.java:94) at com.dtstack.chunjun.sink.DtOutputFormatSinkFunction.open(DtOutputFormatSinkFunction.java:95) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) at org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:46) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:433) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:545) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93) at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:535) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) at java.lang.Thread.run(Thread.java:750)

What you expected to happen

The hive module conflicts with the hive3 module

How to reproduce

bin/start-chunjun -mode local -jobType sync -job sftp-hive.json -chunjunDistDir $CHUNJUN_HOME

sftp example data:

Path|Replication|ModificationTime|AccessTime|PreferredBlockSize|BlocksCount|FileSize|NSQUOTA|DSQUOTA|Permission|UserName|GroupName /|0|2022-07-05 16:42|1970-01-01 08:00|0|0|0|9223372036854775807|-1|drwxrwxr-x+|hdfs|hdfs /tmp|0|2022-06-29 11:44|1970-01-01 08:00|0|0|0|-1|-1|drwxrwxrwx+|hdfs|hdfs /user|0|2022-04-18 22:40|1970-01-01 08:00|0|0|0|-1|-1|drwxrwxr-x+|hdfs|hdfs

sftp-hive.json

{ "job": { "content": [ { "reader":{ "parameter":{ "path":"xxxx", "protocol":"sftp", "port":22, "isFirstLineHeader":true, "host":"ip", "column":[ { "index":0, "type":"string", "name": "Path" }, { "index":1, "type":"string", "name": "Replication" }, { "index":2, "type":"string", "name": "ModificationTime" }, { "index":3, "type":"string", "name": "AccessTime" }, { "index":4, "type":"string", "name": "PreferredBlockSize" }, { "index":5, "type":"string", "name": "BlocksCount" }, { "index":6, "type":"string", "name": "FileSize" }, { "index":7, "type":"string", "name": "NSQUOTA" }, { "index":8, "type":"string", "name": "DSQUOTA" }, { "index":9, "type":"string", "name": "Permission" },
{ "index":10, "type":"string", "name": "UserName" }, { "index":11, "type":"string", "name": "GroupName" }
], "password":"xxxx", "fieldDelimiter":"|", "encoding":"utf-8", "username":"xxxx", "fileType":"txt", "compressType":"" }, "name":"ftpreader" }, "writer": { "name" : "hivewriter", "parameter" : { "jdbcUrl" : "jdbc:hive2://xxx:10000/snl_test", "username":"xxxxx", "password":"xxxxx", "fileType" : "text", "writeMode" : "overwrite", "compress" : "GZIP", "charsetName" : "UTF-8", "tablesColumn" : "{\"fsimage\":[{\"key\":\"Path\",\"type\":\"string\"},{\"key\":\"Replication\",\"type\":\"string\"},{\"key\":\"ModificationTime\",\"type\":\"string\"},{\"key\":\"AccessTime\",\"type\":\"string\"},{\"key\":\"PreferredBlockSize\",\"type\":\"string\"},{\"key\":\"BlocksCount\",\"type\":\"string\"},{\"key\":\"FileSize\",\"type\":\"string\"},{\"key\":\"NSQUOTA\",\"type\":\"string\"},{\"key\":\"DSQUOTA\",\"type\":\"string\"},{\"key\":\"Permission\",\"type\":\"string\"},{\"key\":\"UserName\",\"type\":\"string\"},{\"key\":\"GroupName\",\"type\":\"string\"}]}", "partition" : "pt", "partitionType" : "DAY", "defaultFS" : "hdfs://ns", "hadoopConfig": { "hadoop.user.name": "root", "dfs.ha.namenodes.ns": "nn1,nn2", "fs.defaultFS": "hdfs://ns", "dfs.namenode.rpc-address.ns.nn1": "xx:8020", "dfs.namenode.rpc-address.ns.nn2": "xxx:8020", "dfs.client.failover.proxy.provider.ns": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.nameservices": "ns", "fs.hdfs.impl.disable.cache": "true", "fs.hdfs.impl": "org.apache.hadoop.hdfs.DistributedFileSystem" }

      }
    }
  }
],
"setting": {
  "speed": {
    "bytes": 0,
    "channel": 1
  }
}

} }

Anything else

No response

Version

master

Are you willing to submit PR?

Code of Conduct

yc576192 commented 1 year ago

我也遇到了同样的问题。

qijizong commented 1 year ago

应该是hive版本的问题，我的hive是2.3.7,chunjun-1.12分支hive的依赖为2.1.0，修改hive connector的hive依赖的版本重新编译后解决。希望对你有帮助