search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.11k stars 915 forks source link

[Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() #6799

Closed inkkim closed 5 hours ago

inkkim commented 2 weeks ago

Code of Conduct

Search before asking

Describe the bug

org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress; at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342) at scala.Option.foreach(Option.scala:407) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407) at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

Affects Version(s)

v1.10

Kyuubi Server Log Output

2024-11-06 18:35:04.034 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2024-11-06 18:35:04.034 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=ZOOKE
EPER-001:2181,ZOOKEEPER-002:2181,ZOOKEEPER-003:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@1e19cc16
2024-11-06 18:35:04.035 INFO KyuubiSessionManager-exec-pool: Thread-219-SendThread(ZOOKEEPER-003:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket co
nnection to server ZOOKEEPER-003/10.10.10.10:2181. Will not attempt to authenticate using SASL (unknown error)
2024-11-06 18:35:04.036 INFO KyuubiSessionManager-exec-pool: Thread-219-SendThread(ZOOKEEPER-003:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection
 established to ZOOKEEPER-003/10.10.10.10:2181, initiating session
2024-11-06 18:35:04.037 INFO KyuubiSessionManager-exec-pool: Thread-219-SendThread(ZOOKEEPER-003:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establish
ment complete on server ZOOKEEPER-003/10.10.10.10:2181, sessionid = 0x300231d79b100b7, negotiated timeout = 40000
2024-11-06 18:35:04.038 INFO KyuubiSessionManager-exec-pool: Thread-219-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2024-11-06 18:35:04.044 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.Utils: Loading Kyuubi properties from /usr/local/spark/conf/spark-defaults.conf
2024-11-06 18:35:04.046 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/apache-kyuubi-1.10.0-bin/work/kyuubi/kyuubi-spark-sql
-engine.log.10
2024-11-06 18:35:04.047 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/spark-3.5.1-bin-hadoop3/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.client.ipAddress=127.0.0.1 \
        --conf spark.kyuubi.client.version=1.10.0 \
        --conf spark.kyuubi.engine.appMgrInfo=eyJyZXNvdXJjZU1hbmFnZXIiOiJ5YXJuIiwia3ViZXJuZXRlc0luZm8iOnsiY29udGV4dCI6bnVsbCwibmFtZXNwYWNlIjpudWxsfX0= \
        --conf spark.kyuubi.engine.engineLog.path=/opt/apache-kyuubi-1.10.0-bin/work/kyuubi/kyuubi-spark-sql-engine.log.10 \
        --conf spark.kyuubi.engine.share.level=USER \
        --conf spark.kyuubi.engine.submit.time=1730885704044 \
        --conf spark.kyuubi.engine.type=SPARK_SQL \
        --conf spark.kyuubi.engine.ui.stop.enabled=true \
        --conf spark.kyuubi.frontend.protocols=THRIFT_BINARY,REST \
        --conf spark.kyuubi.ha.addresses=ZOOKEEPER-001:2181,ZOOKEEPER-002:2181,ZOOKEEPER-003:2181 \
        --conf spark.kyuubi.ha.engine.ref.id=66be7e12-49dc-4de1-890f-297c086342a8 \
        --conf spark.kyuubi.ha.namespace=/kyuubi_1.10.0_USER_SPARK_SQL/kyuubi/default \
        --conf spark.kyuubi.operation.result.format=arrow \
        --conf spark.kyuubi.server.ipAddress=10.10.10.10 \
        --conf spark.kyuubi.session.connection.url=10.10.10.10:10009 \
        --conf spark.kyuubi.session.engine.idle.timeout=PT0M \
        --conf spark.kyuubi.session.engine.spark.showProgress=true \
        --conf spark.kyuubi.session.real.user=kyuubi \
        --conf spark.app.name=kyuubi_USER_SPARK_SQL_kyuubi_default_66be7e12-49dc-4de1-890f-297c086342a8 \
        --conf spark.driver.extraJavaOptions=-Daws.region=us-west-2 \
        --conf spark.executor.extraJavaOptions=-Daws.region=us-west-2 \
        --conf spark.sql.catalog.sample=org.apache.iceberg.spark.SparkCatalog \
        --conf spark.sql.catalog.sample.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
        --conf spark.sql.catalog.sample.s3.access-key-id=sample \
        --conf spark.sql.catalog.sample.s3.endpoint=http://10.10.10.10:9020 \
        --conf spark.sql.catalog.sample.s3.path-style-access=true \
        --conf spark.sql.catalog.sample.s3.secret-access-key=********************** \
        --conf spark.sql.catalog.sample.type=rest \
        --conf spark.sql.catalog.sample.uri=http://rest.test.com:9083 \
        --conf spark.sql.catalog.sample.warehouse=s3a://sample/ \
        --conf spark.sql.defaultCatalog=sample \
        --conf spark.yarn.maxAppAttempts=1 \
        --conf spark.yarn.tags=KYUUBI,66be7e12-49dc-4de1-890f-297c086342a8 \
        --proxy-user kyuubi /opt/apache-kyuubi-1.10.0-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.10.0.jar
2024-11-06 18:35:39.450 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2024-11-06 18:35:39.452 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x300231d79b100b7 closed
2024-11-06 18:35:39.452 INFO KyuubiSessionManager-exec-pool: Thread-219-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x300231d79b100b7
2024-11-06 18:35:39.453 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.operation.LaunchEngine: Processing kyuubi's query[c4c29906-fd8b-47d5-8228-b18dcf6a67d6]: RUNNING_STATE -> ERROR_STATE, time taken: 35.419 seconds
2024-11-06 18:35:39.453 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.operation.OperationAuditLogger: operation=c4c29906-fd8b-47d5-8228-b18dcf6a67d6        opType=LaunchEngine  state=ERROR     user=kyuubi   session=66be7e12-49dc-4de1-890f-297c086342a8
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [66be7e12-49dc-4de1-890f-297c086342a8]
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.session.KyuubiSessionManager: kyuubi's KyuubiSessionImpl with SessionHandle [66be7e12-49dc-4de1-890f-297c086342a8] is closed, current opening sessions 0
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.operation.LaunchEngine: Processing kyuubi's query[c4c29906-fd8b-47d5-8228-b18dcf6a67d6]: ERROR_STATE -> CLOSED_STATE, time taken: 35.707 seconds
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.operation.OperationAuditLogger: operation=c4c29906-fd8b-47d5-8228-b18dcf6a67d6     opType=LaunchEngine  state=CLOSED    user=kyuubi   session=66be7e12-49dc-4de1-890f-297c086342a8
2024-11-06 18:35:39.742 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [66be7e12-49dc-4de1-890f-297c086342a8]
2024-11-06 18:36:44.329 ERROR SIGTERM handler org.apache.kyuubi.server.KyuubiServer: RECEIVED SIGNAL 15: TERM
2024-11-06 18:36:44.332 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiRestFrontendService] is stopping.
2024-11-06 18:36:44.335 INFO shutdown-hook-0 org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@34780cd9{HTTP/1.1, (http/1.1)}{10.10.10.10:10099}
2024-11-06 18:36:44.337 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is stopped.
2024-11-06 18:36:44.337 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiTBinaryFrontend] is stopping.
2024-11-06 18:36:44.337 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service: [KyuubiServiceDiscovery] is stopping.
2024-11-06 18:36:44.339 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2024-11-06 18:36:44.339 WARN main-EventThread org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: This Kyuubi instance 10.10.10.10:10009 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
2024-11-06 18:36:44.340 INFO shutdown-hook-0 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x300231d79b1009c closed
2024-11-06 18:36:44.340 INFO shutdown-hook-0 org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is stopped.
2024-11-06 18:36:44.340 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service[KyuubiTBinaryFrontend] is stopped.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: KyuubiTBinaryFrontend has stopped
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiBackendService] is stopping.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiSessionManager] is stopping.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [KyuubiOperationManager] is stopping.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.operation.KyuubiOperationManager: Service[KyuubiOperationManager] is stopped.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [MetadataManager] is stopping.
2024-11-06 18:36:44.342 INFO shutdown-hook-0 com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown initiated...
2024-11-06 18:36:44.344 INFO shutdown-hook-0 com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown completed.
2024-11-06 18:36:44.344 INFO shutdown-hook-0 org.apache.kyuubi.server.metadata.MetadataManager: Service[MetadataManager] is stopped.
2024-11-06 18:36:44.344 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [TempFileService] is stopping.
2024-11-06 18:36:44.345 INFO shutdown-hook-0 org.apache.kyuubi.service.TempFileService: Service[TempFileService] is stopped.
2024-11-06 18:36:44.345 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [HadoopCredentialsManager] is stopping.
2024-11-06 18:36:44.346 INFO shutdown-hook-0 org.apache.kyuubi.credentials.HadoopCredentialsManager: Service[HadoopCredentialsManager] is stopped.
2024-11-06 18:36:44.346 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [KyuubiApplicationManager] is stopping.
2024-11-06 18:36:44.347 INFO shutdown-hook-0 org.apache.kyuubi.engine.KyuubiApplicationManager: Service[KyuubiApplicationManager] is stopped.
2024-11-06 18:36:44.347 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service[KyuubiSessionManager] is stopped.
2024-11-06 18:36:44.352 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service[KyuubiBackendService] is stopped.
2024-11-06 18:36:44.352 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [MetricsSystem] is stopping.
2024-11-06 18:36:44.353 INFO shutdown-hook-0 org.apache.kyuubi.metrics.MetricsSystem: Service: [PrometheusReporterService] is stopping.
2024-11-06 18:36:44.353 INFO shutdown-hook-0 org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@54d18072{HTTP/1.1, (http/1.1)}{0.0.0.0:10019}
2024-11-06 18:36:44.354 INFO shutdown-hook-0 org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@712cfb63{/,null,STOPPED}
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.metrics.PrometheusReporterService: Prometheus metrics HTTP server has stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.metrics.PrometheusReporterService: Service[PrometheusReporterService] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.metrics.MetricsSystem: Service[MetricsSystem] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [PeriodicGCService] is stopping.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.PeriodicGCService: Service[PeriodicGCService] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KinitAuxiliaryService] is stopping.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is stopped.
2024-11-06 18:36:44.355 INFO main-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x300231d79b1009c

Kyuubi Engine Log Output

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:176)
        at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        ... 6 more
Caused by: java.lang.NoSuchMethodError: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
        at org.apache.kyuubi.service.TFrontendService.$anonfun$serverAddr$2(TFrontendService.scala:56)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.kyuubi.service.TFrontendService.serverAddr$lzycompute(TFrontendService.scala:56)
        at org.apache.kyuubi.service.TFrontendService.serverAddr(TFrontendService.scala:55)
        at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:121)
        at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
        at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
        at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:70)
        at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:353)
        ... 22 more

Kyuubi Server Configurations

kyuubi.authentication                    NONE
kyuubi.frontend.bind.host                SPARKYARN-001
kyuubi.frontend.protocols                THRIFT_BINARY,REST
kyuubi.frontend.thrift.binary.bind.port  10009
kyuubi.frontend.rest.bind.port           10099
kyuubi.operation.result.format           arrow
kyuubi.engine.type                       SPARK_SQL
kyuubi.engine.share.level                USER
kyuubi.session.engine.idle.timeout       PT0M
kyuubi.engine.ui.stop.enabled             true
kyuubi.session.engine.spark.showProgress true
kyuubi.ha.addresses                      ZOOKEEPER-001:2181,ZOOKEEPER-002:2181,ZOOKEEPER-003:2181
kyuubi.ha.namespace                      kyuubi

# kyuubi-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export KYUUBI_HOME=/opt/apache-kyuubi-1.10.0-bin
export KYUUBI_ENGINE_MEMORY=4g
export KYUUBI_JAVA_OPTS="-Xmx14g -Xms2g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=4084m  -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:./logs/kyuubi-server-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:NewRatio=3 -XX:MetaspaceSize=1024m"

Kyuubi Engine Configurations

spark.master                     yarn
spark.executor.cores             3
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.driver.memory              3g
spark.executor.memory            10g
spark.yarn.am.memory             1g
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs:///user/spark/history/done
spark.history.fs.logDirectory    hdfs:///user/spark/history/done
spark.worker.cleanup.enabled     true
spark.worker.cleanup.interval    86400
spark.worker.cleanup.appDataTtl  604800
#spark.dynamicAllocation.enabled  false
spark.yarn.queue                 q1
spark.yarn.jars                  file:/usr/local/spark/jars/*
#spark.history.fs.logDirectory    file:/usr/local/spark/logs/spark-events
spark.history.fs.logDirectory    hdfs:///user/spark/history/done
spark.eventLog.compress          true
spark.driver.extraClassPath=$SPARK_HOME/jars/spark-network-yarn_2.12-3.5.1.jar
spark.submit.proxyUser.allowCustomClasspathInClusterMode true
spark.shuffle.compress true
spark.shuffle.spill.compress true
spark.sql.shuffle.partitions 240
spark.rdd.compress     true

Additional context

Here’s a detailed issue description based on the provided information:

Title

Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress()

Description

When attempting to start the Kyuubi Engine using the kyuubi-hive-jdbc-shaded-1.10.0 driver to establish a JDBC Connection, the following error occurs:

The Kyuubi Engine initiates a Spark application on YARN, but it fails shortly after with the following log output:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174)
    ...
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
    at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
    ...
Caused by: java.lang.NoSuchMethodError: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
    at org.apache.kyuubi.service.TFrontendService.$anonfun$serverAddr$2(TFrontendService.scala:56)
    ...

Steps to Reproduce

  1. Start Kyuubi Engine using kyuubi-hive-jdbc-shaded-1.10.0 driver to establish a JDBC Connection.
  2. Observe that a Spark Application is created in YARN.
  3. The process fails, generating the above log output and terminating with an error.

Observed Behavior

Expected Behavior

Environment

Additional Information

This issue might be related to missing or incompatible dependencies in org.apache.kyuubi.util. Any guidance on resolving this error or additional dependencies would be appreciated.

This template should help provide a clear and structured description for others to understand and investigate the issue.

Are you willing to submit PR?

github-actions[bot] commented 2 weeks ago

Hello @inkkim, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.

wForget commented 2 weeks ago

Are you using a self built distribution package?

inkkim commented 2 weeks ago

@wForget Thank you for your response! No, I’m not using a self-built distribution package. I downloaded the Binary directly from the official Kyuubi releases page (Kyuubi Releases) and followed the standard installation steps.

pan3793 commented 2 weeks ago

have you put any jars under /usr/local/spark/jars that are not shipped by the official spark tgz?

inkkim commented 1 week ago

@pan3793 Yes, there are few things. Do I have to remove those before start kyuubi?

pan3793 commented 1 week ago

you should ensure there are no other kyuubi libs that version lower the engine jar.

kyuubi-spark-sql-engine_2.12-1.10.0.jar

for example, you should NOT suppose it works if you put a kyuubi-hive-jdbc-shaded-1.9.3.jar under the $SPARK_HOME/jars

inkkim commented 5 hours ago

@pan3793 Thanks for the clarification! I followed your advice and made sure there were no incompatible Kyuubi libraries, and it worked perfectly without any errors. Specifically, I found kyuubi-spark-connector-tpcds_2.12-1.9.0.jar under /usr/local/spark/jars. Once I removed it and re-ran the setup, everything worked as expected. I really appreciate your guidance!