search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.09k stars 910 forks source link

[Bug] Kyuubi server start failed with NoClassDefFoundError #2101

Closed cfmcgrady closed 2 years ago

cfmcgrady commented 2 years ago

Code of Conduct

Search before asking

Describe the bug

11:03:48.279 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user xxxxx using keytab file xxxxxx. Keytab auto renewal enabled : false
11:03:48.284 INFO org.apache.kyuubi.server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is initialized.
11:03:48.305 INFO org.apache.kyuubi.server.KyuubiEventLoggingService: Service[EventLogging] is initialized.
11:03:48.442 INFO org.apache.kyuubi.metrics.JsonReporterService: Service[JsonReporterService] is initialized.
11:03:48.442 INFO org.apache.kyuubi.metrics.MetricsSystem: Service[MetricsSystem] is initialized.
11:03:48.446 INFO org.apache.kyuubi.util.ThreadUtils: KyuubiSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 60000 ms
11:03:48.987 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
11:03:49.006 INFO org.apache.hadoop.hive.conf.HiveConf: Found configuration file file:/etc/hadoop/yarn-conf/hive-site.xml
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/transport/TFramedTransport
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1708)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:89)
    at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.initialize(HiveDelegationTokenProvider.scala:54)
    at org.apache.kyuubi.credentials.HadoopCredentialsManager.$anonfun$initialize$1(HadoopCredentialsManager.scala:95)
    at org.apache.kyuubi.credentials.HadoopCredentialsManager.$anonfun$initialize$1$adapted(HadoopCredentialsManager.scala:94)
    at scala.collection.immutable.Map$Map2.filterImpl(Map.scala:282)
    at scala.collection.immutable.Map$Map2.filterImpl(Map.scala:222)
    at scala.collection.TraversableLike.filter(TraversableLike.scala:395)
    at scala.collection.TraversableLike.filter$(TraversableLike.scala:395)
    at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
    at org.apache.kyuubi.credentials.HadoopCredentialsManager.initialize(HadoopCredentialsManager.scala:94)
    at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
    at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
    at org.apache.kyuubi.session.SessionManager.initialize(SessionManager.scala:200)
    at org.apache.kyuubi.session.KyuubiSessionManager.initialize(KyuubiSessionManager.scala:45)
    at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
    at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
    at org.apache.kyuubi.service.AbstractBackendService.initialize(AbstractBackendService.scala:174)
    at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
    at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
    at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:46)
    at org.apache.kyuubi.server.KyuubiServer.initialize(KyuubiServer.scala:167)
    at org.apache.kyuubi.server.KyuubiServer$.startServer(KyuubiServer.scala:97)
    at org.apache.kyuubi.server.KyuubiServer$.main(KyuubiServer.scala:133)
    at org.apache.kyuubi.server.KyuubiServer.main(KyuubiServer.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.transport.TFramedTransport
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 40 more

Affects Version(s)

master

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

cfmcgrady commented 2 years ago

cc @pan3793

yaooqinn commented 2 years ago

how about 1.5？

pan3793 commented 2 years ago

kyuubi-server/src/test/java/org/apache/thrift/transport/TFramedTransport.java should be moved from test scope to compile scope

SteNicholas commented 2 years ago

@pan3793 , IMO, the TFramedTransport should be moved to compile scope in the modules which use this. @yaooqinn WDYT?

yaooqinn commented 2 years ago

+1

pan3793 commented 2 years ago

Due to lack of test coverage and up coming 1.5 release, I propose to revert [KYUUBI #1948] Upgrade thrift version to 0.16.0 in branch-1.5, we can continue add the test in master branch in the future.

SteNicholas commented 2 years ago

@pan3793 @yaooqinn, IMO, [KYUUBI #1948] Upgrade thrift version to 0.16.0 doesn't need to revert in branch-1.5. At present, only find this bug for the upgrade. The previous bug of the fb303 has nothing with the upgrade.

cfmcgrady commented 2 years ago

Due to lack of test coverage and up coming 1.5 release, I propose to revert [KYUUBI #1948] Upgrade thrift version to 0.16.0 in branch-1.5, we can continue add the test in master branch in the future.

+1

yaooqinn commented 2 years ago

Technically, shading shall have fixed the issues we have met. But considering Spark(have reverted) and hive stays in 0.9, I am +1 for reverting it temporarily in 1.5 and keeping it in master

cfmcgrady commented 2 years ago

Another error occurs after changing the scope to compile.

Kyuubi server logs ``` 13:22:55.864 INFO org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started. 13:24:26.709 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:202) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.kyuubi.service.authentication.HadoopThriftAuthBridgeServer$TUGIAssumingTransportFactory$$anon$4.run(HadoopThriftAuthBridgeServer.scala:117) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.service.authentication.HadoopThriftAuthBridgeServer$TUGIAssumingTransportFactory$$anon$4.run(HadoopThriftAuthBridgeServer.scala:116) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_171] at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_171] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) ~[hadoop-client-api-3.3.2.jar:?] at org.apache.kyuubi.service.authentication.HadoopThriftAuthBridgeServer$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridgeServer.scala:116) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) ~[libthrift-0.16.0.jar:0.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171] Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:171) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:108) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:238) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:44) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:199) ~[libthrift-0.16.0.jar:0.16.0] ... 10 more 13:24:34.350 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 13:24:34.426 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/c346da28-f388-47ba-bd57-96a6950263ad/49c6d14b-bc8b-4d2b-acbc-2664fcccacd4 13:24:34.453 INFO org.apache.kyuubi.operation.LaunchEngine: Processing chenfu's query[49c6d14b-bc8b-4d2b-acbc-2664fcccacd4]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE 13:24:34.456 INFO org.apache.kyuubi.operation.LaunchEngine: Processing chenfu's query[49c6d14b-bc8b-4d2b-acbc-2664fcccacd4]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE 13:24:34.456 INFO org.apache.kyuubi.session.KyuubiSessionManager: chenfu's session with SessionHandle [c346da28-f388-47ba-bd57-96a6950263ad] is opened, current opening sessions 1 13:24:34.461 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting 13:24:34.461 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=xxxxxx sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@30c47101 13:24:34.465 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server xxxxxx. Will not attempt to authenticate using SASL (unknown error) 13:24:34.466 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to xxxxxx, initiating session 13:24:34.466 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.25.8.213:39680 13:24:34.467 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.25.8.213:39680 13:24:34.469 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1082f42d83a0001 with negotiated timeout 60000 for client /10.25.8.213:39680 13:24:34.469 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server xxxxxxx, sessionid = 0x1082f42d83a0001, negotiated timeout = 60000 13:24:34.469 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED 13:24:35.138 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1082f42d83a0001 type:create cxid:0x2 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/kyuubi_DYNAMIC/lock/chenfu/engine-pool-1/locks Error:KeeperErrorCode = NoNode for /kyuubi_DYNAMIC/lock/chenfu/engine-pool-1/locks 13:24:35.141 WARN org.apache.curator.utils.ZKPaths: The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. 13:24:35.186 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1082f42d83a0001 type:create cxid:0xf zxid:0xc txntype:-1 reqpath:n/a Error Path:/kyuubi_DYNAMIC/lock/chenfu/engine-pool-1/leases Error:KeeperErrorCode = NoNode for /kyuubi_DYNAMIC/lock/chenfu/engine-pool-1/leases 13:24:35.246 INFO org.apache.kyuubi.engine.EngineRef: Launching engine: /xxxxxx/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --proxy-user chenfu /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.6.0-SNAPSHOT.jar 13:24:35.271 INFO org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/kyuubi/work/chenfu/kyuubi-spark-sql-engine.log.0 13:25:06.439 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/c346da28-f388-47ba-bd57-96a6950263ad/1d96bfd8-80ac-47d9-aea3-b46fdadcf4e8 13:25:06.441 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [chenfu:10.25.109.148] SessionHandle [c346da28-f388-47ba-bd57-96a6950263ad] - Starting to wait the launch engine operation finished 13:25:17.305 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.25.44.177:39020 13:25:17.310 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.25.44.177:39020 13:25:17.312 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1082f42d83a0002 with negotiated timeout 60000 for client /10.25.44.177:39020 13:25:17.384 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1082f42d83a0002 type:create cxid:0x1 zxid:0x11 txntype:-1 reqpath:n/a Error Path:/kyuubi_1.6.0-SNAPSHOT_DYNAMIC_SPARK_SQL/chenfu Error:KeeperErrorCode = NoNode for /kyuubi_1.6.0-SNAPSHOT_DYNAMIC_SPARK_SQL/chenfu 13:25:18.361 INFO org.apache.kyuubi.ha.client.ServiceDiscovery: Get service instance:10.25.44.177:45935 and version:Some(1.6.0-SNAPSHOT) under /kyuubi_1.6.0-SNAPSHOT_DYNAMIC_SPARK_SQL/chenfu/engine-pool-1 13:25:18.723 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [chenfu:10.25.109.148] SessionHandle [c346da28-f388-47ba-bd57-96a6950263ad] - Connected to engine [10.25.44.177:45935] with SessionHandle [d17320e9-fb5a-48a1-aaac-38806e0c8c85] 13:25:18.725 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 13:25:18.752 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1082f42d83a0001 13:25:18.753 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1082f42d83a0001 closed 13:25:18.754 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x1082f42d83a0001 13:25:18.755 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.25.8.213:39680 which had sessionid 0x1082f42d83a0001 13:25:18.773 INFO org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider: getting token owned by chenfu for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1698849409_44, ugi=chenfu (auth:PROXY) via xxxxxx (auth:KERBEROS)]] 13:25:18.797 INFO org.apache.hadoop.hdfs.DFSClient: Created token for chenfu: HDFS_DELEGATION_TOKEN owner=chenfu, renewer=chenfu, realUser=xxxxxxx, issueDate=1646976318770, maxDate=1647581118770, sequenceNumber=304199775, masterKeyId=2382 on ha-hdfs:nameservice1 13:25:18.808 INFO org.apache.kyuubi.credentials.HiveDelegationTokenProvider: Getting Hive delegation token for chenfu against xxxxxx 13:25:18.837 INFO org.apache.kyuubi.operation.LaunchEngine: Processing chenfu's query[49c6d14b-bc8b-4d2b-acbc-2664fcccacd4]: RUNNING_STATE -> ERROR_STATE, statement: LAUNCH_ENGINE, time taken: 44.381 seconds 13:25:18.839 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error executing statement: java.util.concurrent.ExecutionException: org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: java.lang.AbstractMethodError at org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444) at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:415) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:251) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_delegation_token(ThriftHiveMetastore.java:4274) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_delegation_token(ThriftHiveMetastore.java:4260) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:2039) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) at com.sun.proxy.$Proxy28.getDelegationToken(Unknown Source) at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HiveDelegationTokenProvider.scala:64) at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.$anonfun$obtainDelegationTokens$1$adapted(HiveDelegationTokenProvider.scala:62) at scala.Option.foreach(Option.scala:407) at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.obtainDelegationTokens(HiveDelegationTokenProvider.scala:62) at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:224) at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:224) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214) at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:224) at org.apache.kyuubi.credentials.HadoopCredentialsManager.org$apache$kyuubi$credentials$HadoopCredentialsManager$$scheduleRenewal(HadoopCredentialsManager.scala:247) at org.apache.kyuubi.credentials.HadoopCredentialsManager.$anonfun$getOrCreateUserCredentialsRef$1(HadoopCredentialsManager.scala:192) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at org.apache.kyuubi.credentials.HadoopCredentialsManager.getOrCreateUserCredentialsRef(HadoopCredentialsManager.scala:190) at org.apache.kyuubi.credentials.HadoopCredentialsManager.sendCredentialsIfNeeded(HadoopCredentialsManager.scala:155) at org.apache.kyuubi.operation.LaunchEngine.renewEngineCredentials(LaunchEngine.scala:71) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:52) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_171] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_171] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$waitForEngineLaunched$1(KyuubiSessionImpl.scala:130) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$waitForEngineLaunched$1$adapted(KyuubiSessionImpl.scala:126) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.session.KyuubiSessionImpl.waitForEngineLaunched(KyuubiSessionImpl.scala:126) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionImpl.runOperation(KyuubiSessionImpl.scala:116) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:123) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:76) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:120) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$executeStatement(KyuubiServer.scala:142) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$executeStatement$1(BackendServiceMetric.scala:62) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.executeStatement(BackendServiceMetric.scala:62) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.executeStatement$(BackendServiceMetric.scala:55) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.executeStatement(KyuubiServer.scala:142) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:232) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) ~[hive-service-rpc-3.1.2.jar:3.1.2] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) ~[hive-service-rpc-3.1.2.jar:3.1.2] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.kyuubi.service.authentication.HadoopThriftAuthBridgeServer$TUGIAssumingProcessor.process(HadoopThriftAuthBridgeServer.scala:163) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_171] Caused by: org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: java.lang.AbstractMethodError at org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444) at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:415) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:251) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_delegation_token(ThriftHiveMetastore.java:4274) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_delegation_token(ThriftHiveMetastore.java:4260) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:2039) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) at com.sun.proxy.$Proxy28.getDelegationToken(Unknown Source) at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HiveDelegationTokenProvider.scala:64) at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.$anonfun$obtainDelegationTokens$1$adapted(HiveDelegationTokenProvider.scala:62) at scala.Option.foreach(Option.scala:407) at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.obtainDelegationTokens(HiveDelegationTokenProvider.scala:62) at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:224) at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:224) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214) at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:224) at org.apache.kyuubi.credentials.HadoopCredentialsManager.org$apache$kyuubi$credentials$HadoopCredentialsManager$$scheduleRenewal(HadoopCredentialsManager.scala:247) at org.apache.kyuubi.credentials.HadoopCredentialsManager.$anonfun$getOrCreateUserCredentialsRef$1(HadoopCredentialsManager.scala:192) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at org.apache.kyuubi.credentials.HadoopCredentialsManager.getOrCreateUserCredentialsRef(HadoopCredentialsManager.scala:190) at org.apache.kyuubi.credentials.HadoopCredentialsManager.sendCredentialsIfNeeded(HadoopCredentialsManager.scala:155) at org.apache.kyuubi.operation.LaunchEngine.renewEngineCredentials(LaunchEngine.scala:71) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:52) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:81) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:62) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:54) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_171] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_171] ... 3 more Caused by: java.lang.AbstractMethodError at org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:415) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:251) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[libthrift-0.16.0.jar:0.16.0] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_delegation_token(ThriftHiveMetastore.java:4274) ~[hive-metastore-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_delegation_token(ThriftHiveMetastore.java:4260) ~[hive-metastore-2.3.9.jar:2.3.9] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDelegationToken(HiveMetaStoreClient.java:2039) ~[hive-metastore-2.3.9.jar:2.3.9] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) ~[hive-metastore-2.3.9.jar:2.3.9] at com.sun.proxy.$Proxy28.getDelegationToken(Unknown Source) ~[?:?] at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HiveDelegationTokenProvider.scala:64) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.$anonfun$obtainDelegationTokens$1$adapted(HiveDelegationTokenProvider.scala:62) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.credentials.HiveDelegationTokenProvider.obtainDelegationTokens(HiveDelegationTokenProvider.scala:62) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:224) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:224) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at scala.collection.Iterator.foreach(Iterator.scala:943) ~[scala-library-2.12.15.jar:?] at scala.collection.Iterator.foreach$(Iterator.scala:943) ~[scala-library-2.12.15.jar:?] at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) ~[scala-library-2.12.15.jar:?] at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:224) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.credentials.HadoopCredentialsManager.org$apache$kyuubi$credentials$HadoopCredentialsManager$$scheduleRenewal(HadoopCredentialsManager.scala:247) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.credentials.HadoopCredentialsManager.$anonfun$getOrCreateUserCredentialsRef$1(HadoopCredentialsManager.scala:192) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_171] at org.apache.kyuubi.credentials.HadoopCredentialsManager.getOrCreateUserCredentialsRef(HadoopCredentialsManager.scala:190) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.credentials.HadoopCredentialsManager.sendCredentialsIfNeeded(HadoopCredentialsManager.scala:155) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.operation.LaunchEngine.renewEngineCredentials(LaunchEngine.scala:71) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:52) ~[kyuubi-server_2.12-1.6.0-SNAPSHOT.jar:1.6.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_171] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_171] ... 3 more ```
SteNicholas commented 2 years ago

@pan3793 @yaooqinn , does the hive-service-rpc version need to downgrade?

pan3793 commented 2 years ago

No, hive-service-rpc 3.1 is compatible with thrift 0.9

pan3793 commented 2 years ago

Looks like we need to revert on master branch due to the last issue reported by @cfmcgrady