search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.09k stars 910 forks source link

[Bug] duplic engine create when use jdbc #2867

Closed lordk911 closed 2 years ago

lordk911 commented 2 years ago

Code of Conduct

Search before asking

Describe the bug

when try to connect to kyuubi ues hive jdbc driver , got duplic engine, after query finish, one engion will ramain on yarn. I'm not use ha model for test

image

Affects Version(s)

1.4.1

Kyuubi Server Log Output

[bigtop@worker-10-0-161-25 apache-kyuubi-1.4.1-incubating-bin]$ bin/kyuubi run
Using kyuubi environment file /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/conf/kyuubi-env.sh to initialize...
JAVA_HOME: /usr/java/jdk1.8.0_281-amd64/
KYUUBI_HOME: /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin
KYUUBI_CONF_DIR: /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/conf
KYUUBI_LOG_DIR: /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/logs
KYUUBI_PID_DIR: /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/pid
KYUUBI_WORK_DIR_ROOT: /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/work
SPARK_HOME: /data/soft/spark3
SPARK_CONF_DIR: /data/soft/spark3/conf
HADOOP_CONF_DIR: /etc/hadoop/conf/
Starting org.apache.kyuubi.server.KyuubiServer
22/06/13 15:00:03 INFO server.KyuubiServer: 
                  Welcome to
  __  __                           __
 /\ \/\ \                         /\ \      __
 \ \ \/'/'  __  __  __  __  __  __\ \ \____/\_\
  \ \ , <  /\ \/\ \/\ \/\ \/\ \/\ \\ \ '__`\/\ \
   \ \ \\`\\ \ \_\ \ \ \_\ \ \ \_\ \\ \ \L\ \ \ \
    \ \_\ \_\/`____ \ \____/\ \____/ \ \_,__/\ \_\
     \/_/\/_/`/___/> \/___/  \/___/   \/___/  \/_/
                /\___/
                \/__/

22/06/13 15:00:03 INFO server.KyuubiServer: Version: 1.4.1-incubating, Revision: 3b9cb0061c70602fbc582bb28e0d59916b43cba7, Branch: branch-1.4, Java: 1.8, Scala: 2.12, Spark: 3.1.2, Hadoop: 3.3.1, Hive: 2.3.9
22/06/13 15:00:03 INFO server.KyuubiServer: Using Scala version 2.12.15, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_281
22/06/13 15:00:03 INFO util.SignalRegister: Registering signal handler for TERM
22/06/13 15:00:03 INFO util.SignalRegister: Registering signal handler for HUP
22/06/13 15:00:03 INFO util.SignalRegister: Registering signal handler for INT
22/06/13 15:00:04 INFO kyuubi.Utils: Loading Kyuubi properties from /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/conf/kyuubi-defaults.conf
22/06/13 15:00:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:host.name=worker-10-0-161-25
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.version=1.8.0_281
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.home=/usr/java/jdk1.8.0_281-amd64/jre
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.class.path=/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient_common-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-module-paranamer-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.xml.bind-api-2.3.2.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-servlet-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-jaxrs2-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-jaxrs-json-provider-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-ui-4.1.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/slf4j-log4j12-1.7.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/curator-framework-2.12.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/curator-client-2.12.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/htrace-core4-4.1.0-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-common-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-zookeeper_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jline-0.9.94.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/slf4j-api-1.7.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/libthrift-0.9.3.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-common-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-scala-module_2.12-2.5.2.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/javassist-3.25.0-GA.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/zookeeper-3.4.14.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-http-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-media-jaxb-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-core-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-integration-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-jvm-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-jaxrs-base-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/log4j-1.2.17.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jcl-over-slf4j-1.7.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-service-rpc-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hk2-locator-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-ctl_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-jmx-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hadoop-client-runtime-3.3.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.ws.rs-api-2.1.6.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-ha_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/aopalliance-repackaged-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-security-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/failureaccess-1.0.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-core-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-models-jakarta-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-server-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-models-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-core-jakarta-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/netty-all-4.1.68.Final.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-json-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/snakeyaml-1.26.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/paranamer-2.8.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jaxb-api-2.2.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-util-ajax-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.activation-api-1.2.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-module-scala_2.12-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-annotations-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-metastore-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-databind-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-container-servlet-core-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/osgi-resource-locator-1.0.3.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/scopt_2.12-4.0.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.inject-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hk2-utils-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-hk2-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.servlet-api-4.0.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/guava-30.1-jre.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-datatype-jsr310-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hadoop-client-api-3.3.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/libfb303-0.9.3.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-module-jaxb-annotations-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/commons-codec-1.15.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hk2-api-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-metrics_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient_dropwizard-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-serde-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/scala-library-2.12.15.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-shims-common-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.annotation-api-1.3.5.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-io-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-annotations-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient_servlet-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-annotations-jakarta-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/commons-lang-2.6.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-media-json-jackson-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-common_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-client-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/commons-lang3-3.10.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-entity-filtering-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-server_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-dataformat-yaml-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/curator-recipes-2.12.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.validation-api-2.0.2.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-shims-0.23-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-core-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-server-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-util-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/classgraph-4.8.95.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/conf:/etc/hadoop/conf/
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:java.compiler=<NA>
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:os.name=Linux
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:os.arch=amd64
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:os.version=3.10.0-1127.el7.x86_64
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:user.name=bigtop
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:user.home=/home/bigtop
22/06/13 15:00:04 INFO server.ZooKeeperServer: Server environment:user.dir=/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin
22/06/13 15:00:04 INFO server.ZooKeeperServer: Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir embedded_zookeeper/version-2 snapdir embedded_zookeeper/version-2
22/06/13 15:00:04 INFO server.ZooKeeperServer: minSessionTimeout set to 6000
22/06/13 15:00:04 INFO server.ZooKeeperServer: maxSessionTimeout set to 60000
22/06/13 15:00:04 INFO server.NIOServerCnxnFactory: binding to port worker-10-0-161-25/10.0.161.25:2189
22/06/13 15:00:04 INFO zookeeper.EmbeddedZookeeper: Service[EmbeddedZookeeper] is initialized.
22/06/13 15:00:04 INFO zookeeper.EmbeddedZookeeper: EmbeddedZookeeper is started at worker-10-0-161-25:2189
22/06/13 15:00:04 INFO zookeeper.EmbeddedZookeeper: Service[EmbeddedZookeeper] is started.
22/06/13 15:00:04 INFO server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is initialized.
22/06/13 15:00:04 INFO server.EventLoggingService: Service[EventLogging] is initialized.
22/06/13 15:00:04 INFO metrics.JsonReporterService: Service[JsonReporterService] is initialized.
22/06/13 15:00:04 INFO metrics.MetricsSystem: Service[MetricsSystem] is initialized.
22/06/13 15:00:04 INFO util.ThreadUtils: KyuubiSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 60000 ms
22/06/13 15:00:04 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
22/06/13 15:00:05 WARN credentials.HadoopCredentialsManager: Service hadoopfs does not require a token. Check your configuration to see if security is disabled or not.
22/06/13 15:00:05 INFO conf.HiveConf: Found configuration file null
22/06/13 15:00:05 WARN credentials.HadoopCredentialsManager: Service hive does not require a token. Check your configuration to see if security is disabled or not.
22/06/13 15:00:05 WARN credentials.HadoopCredentialsManager: No delegation token is required by services.
22/06/13 15:00:05 INFO credentials.HadoopCredentialsManager: Service[HadoopCredentialsManager] is initialized.
22/06/13 15:00:05 INFO operation.KyuubiOperationManager: Service[KyuubiOperationManager] is initialized.
22/06/13 15:00:05 INFO session.KyuubiSessionManager: Service[KyuubiSessionManager] is initialized.
22/06/13 15:00:05 INFO server.KyuubiBackendService: Service[KyuubiBackendService] is initialized.
22/06/13 15:00:05 INFO server.KyuubiThriftBinaryFrontendService: Initializing KyuubiThriftBinaryFrontendService on host worker-10-0-161-25 at port 10010 with [9, 999] worker threads
22/06/13 15:00:05 INFO imps.CuratorFrameworkImpl: Starting
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:host.name=worker-10-0-161-25
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_281
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_281-amd64/jre
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient_common-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-module-paranamer-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.xml.bind-api-2.3.2.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-servlet-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-jaxrs2-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-jaxrs-json-provider-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-ui-4.1.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/slf4j-log4j12-1.7.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/curator-framework-2.12.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/curator-client-2.12.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/htrace-core4-4.1.0-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-common-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-zookeeper_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jline-0.9.94.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/slf4j-api-1.7.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/libthrift-0.9.3.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-common-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-scala-module_2.12-2.5.2.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/javassist-3.25.0-GA.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/zookeeper-3.4.14.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-http-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-media-jaxb-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-core-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-integration-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-jvm-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-jaxrs-base-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/log4j-1.2.17.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jcl-over-slf4j-1.7.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-service-rpc-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hk2-locator-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-ctl_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-jmx-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hadoop-client-runtime-3.3.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.ws.rs-api-2.1.6.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-ha_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/aopalliance-repackaged-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-security-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/failureaccess-1.0.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-core-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-models-jakarta-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-server-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-models-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-core-jakarta-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/netty-all-4.1.68.Final.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/metrics-json-4.1.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/snakeyaml-1.26.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/paranamer-2.8.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jaxb-api-2.2.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-util-ajax-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.activation-api-1.2.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-module-scala_2.12-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-annotations-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-metastore-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-databind-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-container-servlet-core-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/osgi-resource-locator-1.0.3.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/scopt_2.12-4.0.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.inject-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hk2-utils-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-hk2-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.servlet-api-4.0.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/guava-30.1-jre.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-datatype-jsr310-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hadoop-client-api-3.3.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/libfb303-0.9.3.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-module-jaxb-annotations-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/commons-codec-1.15.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hk2-api-2.6.1.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-metrics_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient_dropwizard-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-serde-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/scala-library-2.12.15.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-shims-common-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.annotation-api-1.3.5.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-io-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-annotations-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/simpleclient_servlet-0.10.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-annotations-jakarta-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/commons-lang-2.6.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-media-json-jackson-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-common_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-client-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/commons-lang3-3.10.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jersey-entity-filtering-2.30.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/kyuubi-server_2.12-1.4.1-incubating.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jackson-dataformat-yaml-2.11.4.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/curator-recipes-2.12.0.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jakarta.validation-api-2.0.2.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/hive-shims-0.23-2.3.9.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/swagger-core-2.1.11.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-server-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/jetty-util-9.4.41.v20210516.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/jars/classgraph-4.8.95.jar:/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/conf:/etc/hadoop/conf/
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.el7.x86_64
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:user.name=bigtop
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/bigtop
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Client environment:user.dir=/data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin
22/06/13 15:00:05 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=worker-10-0-161-25:2189 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@28f8e165
22/06/13 15:00:05 INFO client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is initialized.
22/06/13 15:00:05 INFO server.KyuubiThriftBinaryFrontendService: Service[KyuubiThriftBinaryFrontendService] is initialized.
22/06/13 15:00:05 INFO server.KyuubiServer: Service[KyuubiServer] is initialized.
22/06/13 15:00:05 INFO server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is started.
22/06/13 15:00:05 INFO server.EventLoggingService: Service[EventLogging] is started.
22/06/13 15:00:05 INFO zookeeper.ClientCnxn: Opening socket connection to server worker-10-0-161-25/10.0.161.25:2189. Will not attempt to authenticate using SASL (unknown error)
22/06/13 15:00:05 INFO metrics.JsonReporterService: Service[JsonReporterService] is started.
22/06/13 15:00:05 INFO metrics.MetricsSystem: Service[MetricsSystem] is started.
22/06/13 15:00:05 INFO credentials.HadoopCredentialsManager: Service[HadoopCredentialsManager] is started.
22/06/13 15:00:05 INFO operation.KyuubiOperationManager: Service[KyuubiOperationManager] is started.
22/06/13 15:00:05 INFO session.KyuubiSessionManager: Service[KyuubiSessionManager] is started.
22/06/13 15:00:05 INFO server.KyuubiBackendService: Service[KyuubiBackendService] is started.
22/06/13 15:00:05 INFO zookeeper.ClientCnxn: Socket connection established to worker-10-0-161-25/10.0.161.25:2189, initiating session
22/06/13 15:00:05 INFO server.NIOServerCnxnFactory: Accepted socket connection from /10.0.161.25:38774
22/06/13 15:00:05 INFO server.ZooKeeperServer: Client attempting to establish new session at /10.0.161.25:38774
22/06/13 15:00:05 INFO persistence.FileTxnLog: Creating new log file: log.1
22/06/13 15:00:05 INFO server.ZooKeeperServer: Established session 0x107871f7cf70000 with negotiated timeout 60000 for client /10.0.161.25:38774
22/06/13 15:00:05 INFO zookeeper.ClientCnxn: Session establishment complete on server worker-10-0-161-25/10.0.161.25:2189, sessionid = 0x107871f7cf70000, negotiated timeout = 60000
22/06/13 15:00:05 INFO state.ConnectionStateManager: State change: CONNECTED
22/06/13 15:00:05 INFO client.KyuubiServiceDiscovery: Zookeeper client connection state changed to: CONNECTED
22/06/13 15:00:05 INFO client.ServiceDiscovery: Created a /kyuubi/serviceUri=worker-10-0-161-25:10010;version=1.4.1-incubating;sequence=0000000000 on ZooKeeper for KyuubiServer uri: worker-10-0-161-25:10010
22/06/13 15:00:05 INFO client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is started.
22/06/13 15:00:05 INFO server.KyuubiThriftBinaryFrontendService: Service[KyuubiThriftBinaryFrontendService] is started.
22/06/13 15:00:05 INFO server.KyuubiServer: Service[KyuubiServer] is started.
22/06/13 15:00:05 INFO server.KyuubiThriftBinaryFrontendService: Starting and exposing JDBC connection at: jdbc:hive2://worker-10-0-161-25:10010/
22/06/13 15:00:40 INFO server.KyuubiThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
22/06/13 15:00:40 INFO log.OperationLog: Creating operation log file /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/work/server_operation_logs/8c2d60b9-b208-4f23-97ec-9137f3acf035/2519daf8-c4b6-4b6c-8052-4d7a9e5fb086
22/06/13 15:00:40 INFO operation.LaunchEngine: Processing bigtop's query[2519daf8-c4b6-4b6c-8052-4d7a9e5fb086]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE
22/06/13 15:00:40 INFO session.KyuubiSessionManager: bigtop's session with SessionHandle [8c2d60b9-b208-4f23-97ec-9137f3acf035] is opened, current opening sessions 1
22/06/13 15:00:40 INFO operation.LaunchEngine: Processing bigtop's query[2519daf8-c4b6-4b6c-8052-4d7a9e5fb086]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE
22/06/13 15:00:40 INFO imps.CuratorFrameworkImpl: Starting
22/06/13 15:00:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=worker-10-0-161-25:2189 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1e483f24
22/06/13 15:00:40 INFO zookeeper.ClientCnxn: Opening socket connection to server worker-10-0-161-25/10.0.161.25:2189. Will not attempt to authenticate using SASL (unknown error)
22/06/13 15:00:40 INFO zookeeper.ClientCnxn: Socket connection established to worker-10-0-161-25/10.0.161.25:2189, initiating session
22/06/13 15:00:40 INFO server.NIOServerCnxnFactory: Accepted socket connection from /10.0.161.25:38872
22/06/13 15:00:40 INFO server.ZooKeeperServer: Client attempting to establish new session at /10.0.161.25:38872
22/06/13 15:00:40 INFO server.ZooKeeperServer: Established session 0x107871f7cf70001 with negotiated timeout 60000 for client /10.0.161.25:38872
22/06/13 15:00:40 INFO zookeeper.ClientCnxn: Session establishment complete on server worker-10-0-161-25/10.0.161.25:2189, sessionid = 0x107871f7cf70001, negotiated timeout = 60000
22/06/13 15:00:40 INFO state.ConnectionStateManager: State change: CONNECTED
22/06/13 15:00:40 INFO engine.EngineRef: Launching engine:
/data/soft/spark-3.2.1-bin-hadoop3.2/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.kyuubi.session.engine.initialize.timeout=PT5M \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.authentication.ldap.url=ldap://10.0.105.246:389/ \
        --conf spark.driver.maxResultSize=5g \
        --conf spark.kyuubi.ha.zookeeper.quorum=worker-10-0-161-25:2189 \
        --conf spark.shuffle.service.enabled=true \
        --conf spark.kyuubi.zookeeper.embedded.client.port=2189 \
        --conf spark.kyuubi.engine.single.spark.session=true \
        --conf spark.kyuubi.operation.incremental.collect=true \
        --conf spark.yarn.queue=query \
        --conf spark.kyuubi.authentication.ldap.base.dn=ou=People,dc=xx,dc=com \
        --conf spark.kyuubi.session.engine.launch.async=true \
        --conf spark.app.name=kyuubi_CONNECTION_SPARK_SQL_bigtop_bd18d30a-8ba1-4f5d-84e1-b602b928e3fa \
        --conf spark.driver.memory=10g \
        --conf spark.executor.instances=2 \
        --conf spark.kyuubi.ha.engine.ref.id=bd18d30a-8ba1-4f5d-84e1-b602b928e3fa \
        --conf spark.dynamicAllocation.maxExecutors=6 \
        --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
        --conf spark.yarn.tags=KYUUBI \
        --conf spark.kyuubi.engine.share.level=CONNECTION \
        --conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_CONNECTION_SPARK_SQL/bigtop/bd18d30a-8ba1-4f5d-84e1-b602b928e3fa \
        --conf spark.kyuubi.authentication.ldap.guidKey=cn \
        --conf spark.executor.memory=4g \
        --conf spark.dynamicAllocation.enabled=true \
        --conf spark.executor.cores=1 \
        --conf spark.dynamicAllocation.minExecutors=0 \
        --conf spark.kyuubi.zookeeper.embedded.port=2189 \
        --conf spark.kyuubi.engine.connection.url.use.hostname=false \
        --proxy-user bigtop /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.1-incubating.jar
22/06/13 15:00:40 INFO engine.ProcBuilder: Logging to /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/work/bigtop/kyuubi-spark-sql-engine.log.9
22/06/13 15:00:40 INFO server.KyuubiThriftBinaryFrontendService: Received request of closing SessionHandle [8c2d60b9-b208-4f23-97ec-9137f3acf035]
22/06/13 15:00:40 INFO session.KyuubiSessionManager: SessionHandle [8c2d60b9-b208-4f23-97ec-9137f3acf035] is closed, current opening sessions 0
22/06/13 15:00:40 INFO operation.LaunchEngine: Processing bigtop's query[2519daf8-c4b6-4b6c-8052-4d7a9e5fb086]: RUNNING_STATE -> CLOSED_STATE, statement: LAUNCH_ENGINE
22/06/13 15:00:40 INFO server.KyuubiThriftBinaryFrontendService: Finished closing SessionHandle [8c2d60b9-b208-4f23-97ec-9137f3acf035]
22/06/13 15:00:40 INFO server.KyuubiThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
22/06/13 15:00:40 INFO log.OperationLog: Creating operation log file /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/work/server_operation_logs/f3bce103-3bea-42c1-a6a1-40aaf725f4a2/f545398c-bd77-429f-b4c2-a78fec960aba
22/06/13 15:00:40 INFO operation.LaunchEngine: Processing bigtop's query[f545398c-bd77-429f-b4c2-a78fec960aba]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE
22/06/13 15:00:40 INFO session.KyuubiSessionManager: bigtop's session with SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2] is opened, current opening sessions 1
22/06/13 15:00:40 INFO operation.LaunchEngine: Processing bigtop's query[f545398c-bd77-429f-b4c2-a78fec960aba]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE
22/06/13 15:00:40 INFO imps.CuratorFrameworkImpl: Starting
22/06/13 15:00:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=worker-10-0-161-25:2189 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@37ab25c9
22/06/13 15:00:40 INFO zookeeper.ClientCnxn: Opening socket connection to server worker-10-0-161-25/10.0.161.25:2189. Will not attempt to authenticate using SASL (unknown error)
22/06/13 15:00:40 INFO zookeeper.ClientCnxn: Socket connection established to worker-10-0-161-25/10.0.161.25:2189, initiating session
22/06/13 15:00:40 INFO server.NIOServerCnxnFactory: Accepted socket connection from /10.0.161.25:38876
22/06/13 15:00:40 INFO server.ZooKeeperServer: Client attempting to establish new session at /10.0.161.25:38876
22/06/13 15:00:40 INFO server.ZooKeeperServer: Established session 0x107871f7cf70002 with negotiated timeout 60000 for client /10.0.161.25:38876
22/06/13 15:00:40 INFO zookeeper.ClientCnxn: Session establishment complete on server worker-10-0-161-25/10.0.161.25:2189, sessionid = 0x107871f7cf70002, negotiated timeout = 60000
22/06/13 15:00:40 INFO state.ConnectionStateManager: State change: CONNECTED
22/06/13 15:00:40 INFO engine.EngineRef: Launching engine:
/data/soft/spark-3.2.1-bin-hadoop3.2/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.kyuubi.session.engine.initialize.timeout=PT5M \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.authentication.ldap.url=ldap://10.0.105.246:389/ \
        --conf spark.driver.maxResultSize=5g \
        --conf spark.kyuubi.ha.zookeeper.quorum=worker-10-0-161-25:2189 \
        --conf spark.shuffle.service.enabled=true \
        --conf spark.kyuubi.zookeeper.embedded.client.port=2189 \
        --conf spark.kyuubi.engine.single.spark.session=true \
        --conf spark.kyuubi.operation.incremental.collect=true \
        --conf spark.yarn.queue=query \
        --conf spark.kyuubi.authentication.ldap.base.dn=ou=People,dc=xx,dc=com \
        --conf spark.kyuubi.session.engine.launch.async=true \
        --conf spark.app.name=kyuubi_CONNECTION_SPARK_SQL_bigtop_596b7256-5790-4dd4-8c6a-50b9b7f6c082 \
        --conf spark.driver.memory=10g \
        --conf spark.executor.instances=2 \
        --conf spark.kyuubi.ha.engine.ref.id=596b7256-5790-4dd4-8c6a-50b9b7f6c082 \
        --conf spark.dynamicAllocation.maxExecutors=6 \
        --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
        --conf spark.yarn.tags=KYUUBI \
        --conf spark.kyuubi.engine.share.level=CONNECTION \
        --conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_CONNECTION_SPARK_SQL/bigtop/596b7256-5790-4dd4-8c6a-50b9b7f6c082 \
        --conf spark.kyuubi.authentication.ldap.guidKey=cn \
        --conf spark.executor.memory=4g \
        --conf spark.dynamicAllocation.enabled=true \
        --conf spark.executor.cores=1 \
        --conf spark.dynamicAllocation.minExecutors=0 \
        --conf spark.kyuubi.zookeeper.embedded.port=2189 \
        --conf spark.kyuubi.engine.connection.url.use.hostname=false \
        --proxy-user bigtop /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.1-incubating.jar
22/06/13 15:00:40 INFO engine.ProcBuilder: Logging to /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/work/bigtop/kyuubi-spark-sql-engine.log.10
22/06/13 15:00:40 INFO log.OperationLog: Creating operation log file /data/soft/kyuubi/apache-kyuubi-1.4.1-incubating-bin/work/server_operation_logs/f3bce103-3bea-42c1-a6a1-40aaf725f4a2/1579575b-4b5a-47db-91f0-b1628fa4eef6
22/06/13 15:00:40 INFO session.KyuubiSessionImpl: [bigtop:10.0.68.165] SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2] - Starting to wait the launch engine operation finished
22/06/13 15:01:03 INFO server.NIOServerCnxnFactory: Accepted socket connection from /10.0.161.23:40388
22/06/13 15:01:03 INFO server.ZooKeeperServer: Client attempting to establish new session at /10.0.161.23:40388
22/06/13 15:01:03 INFO server.ZooKeeperServer: Established session 0x107871f7cf70003 with negotiated timeout 60000 for client /10.0.161.23:40388
22/06/13 15:01:03 INFO server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x107871f7cf70003 type:create cxid:0x1 zxid:0x7 txntype:-1 reqpath:n/a Error Path:/kyuubi_CONNECTION_SPARK_SQL/bigtop Error:KeeperErrorCode = NoNode for /kyuubi_CONNECTION_SPARK_SQL/bigtop
22/06/13 15:01:03 INFO client.ServiceDiscovery: Get service instance:10.0.161.23:46795 and version:Some(1.4.1-incubating) under /kyuubi_CONNECTION_SPARK_SQL/bigtop/bd18d30a-8ba1-4f5d-84e1-b602b928e3fa
22/06/13 15:01:03 INFO session.KyuubiSessionImpl: [bigtop:10.0.68.165] SessionHandle [8c2d60b9-b208-4f23-97ec-9137f3acf035] - Connected to engine [10.0.161.23:46795]
22/06/13 15:01:03 INFO session.KyuubiSessionImpl: [bigtop:10.0.68.165] SessionHandle [8c2d60b9-b208-4f23-97ec-9137f3acf035] - Opened engine session[SessionHandle [42c39f6a-e601-42ce-b4c2-84c9186a267a]]
22/06/13 15:01:03 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
22/06/13 15:01:03 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x107871f7cf70001
22/06/13 15:01:03 INFO zookeeper.ZooKeeper: Session: 0x107871f7cf70001 closed
22/06/13 15:01:03 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x107871f7cf70001
22/06/13 15:01:03 INFO server.NIOServerCnxn: Closed socket connection for client /10.0.161.25:38872 which had sessionid 0x107871f7cf70001
22/06/13 15:01:03 WARN operation.LaunchEngine: Ignore exception in terminal state with 2519daf8-c4b6-4b6c-8052-4d7a9e5fb086: org.apache.kyuubi.KyuubiSQLException: Illegal Operation state transition from CLOSED to FINISHED
22/06/13 15:01:03 INFO server.NIOServerCnxnFactory: Accepted socket connection from /10.0.161.21:52088
22/06/13 15:01:03 INFO server.ZooKeeperServer: Client attempting to establish new session at /10.0.161.21:52088
22/06/13 15:01:03 INFO server.ZooKeeperServer: Established session 0x107871f7cf70004 with negotiated timeout 60000 for client /10.0.161.21:52088
22/06/13 15:01:03 INFO client.ServiceDiscovery: Get service instance:10.0.161.21:40212 and version:Some(1.4.1-incubating) under /kyuubi_CONNECTION_SPARK_SQL/bigtop/596b7256-5790-4dd4-8c6a-50b9b7f6c082
22/06/13 15:01:03 INFO session.KyuubiSessionImpl: [bigtop:10.0.68.165] SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2] - Connected to engine [10.0.161.21:40212]
22/06/13 15:01:04 INFO session.KyuubiSessionImpl: [bigtop:10.0.68.165] SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2] - Opened engine session[SessionHandle [38746d8b-f10c-40d9-8d25-87c95d43ced4]]
22/06/13 15:01:04 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
22/06/13 15:01:04 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x107871f7cf70002
22/06/13 15:01:04 INFO zookeeper.ZooKeeper: Session: 0x107871f7cf70002 closed
22/06/13 15:01:04 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x107871f7cf70002
22/06/13 15:01:04 INFO server.NIOServerCnxn: Closed socket connection for client /10.0.161.25:38876 which had sessionid 0x107871f7cf70002
22/06/13 15:01:04 INFO operation.LaunchEngine: Processing bigtop's query[f545398c-bd77-429f-b4c2-a78fec960aba]: RUNNING_STATE -> FINISHED_STATE, statement: LAUNCH_ENGINE, time taken: 23.273 seconds
22/06/13 15:01:04 INFO session.KyuubiSessionImpl: [bigtop:10.0.68.165] SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2] - Engine has been launched, elapsed time: 23 s
22/06/13 15:01:04 INFO operation.ExecuteStatement: Processing bigtop's query[1579575b-4b5a-47db-91f0-b1628fa4eef6]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT * from stream.aplog limit 10000
22/06/13 15:01:04 INFO operation.ExecuteStatement: Processing bigtop's query[1579575b-4b5a-47db-91f0-b1628fa4eef6]: PENDING_STATE -> RUNNING_STATE, statement: SELECT * from stream.aplog limit 10000
22/06/13 15:01:09 INFO operation.ExecuteStatement: Query[1579575b-4b5a-47db-91f0-b1628fa4eef6] in RUNNING_STATE
22/06/13 15:01:14 INFO operation.ExecuteStatement: Query[1579575b-4b5a-47db-91f0-b1628fa4eef6] in RUNNING_STATE
22/06/13 15:01:18 INFO operation.ExecuteStatement: Query[1579575b-4b5a-47db-91f0-b1628fa4eef6] in FINISHED_STATE
22/06/13 15:01:18 INFO operation.ExecuteStatement: Processing bigtop's query[1579575b-4b5a-47db-91f0-b1628fa4eef6]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT * from stream.aplog limit 10000, time taken: 14.776 seconds
22/06/13 15:01:22 INFO server.KyuubiThriftBinaryFrontendService: Received request of closing SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2]
22/06/13 15:01:22 INFO session.KyuubiSessionManager: SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2] is closed, current opening sessions 0
22/06/13 15:01:22 INFO operation.LaunchEngine: Processing bigtop's query[f545398c-bd77-429f-b4c2-a78fec960aba]: FINISHED_STATE -> CLOSED_STATE, statement: LAUNCH_ENGINE
22/06/13 15:01:22 INFO operation.ExecuteStatement: Processing bigtop's query[1579575b-4b5a-47db-91f0-b1628fa4eef6]: FINISHED_STATE -> CLOSED_STATE, statement: SELECT * from stream.aplog limit 10000
22/06/13 15:01:22 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:E3 B0 4A 8E 0E 9E 48 D1 85 10 11 63 4C 8C AC 39, secret:C3 0F 2D CF C2 44 45 90 BF 69 8C 63 EA DD E2 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
22/06/13 15:01:22 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x107871f7cf70004
22/06/13 15:01:22 INFO server.NIOServerCnxn: Closed socket connection for client /10.0.161.21:52088 which had sessionid 0x107871f7cf70004
22/06/13 15:01:22 INFO server.KyuubiThriftBinaryFrontendService: Finished closing SessionHandle [f3bce103-3bea-42c1-a6a1-40aaf725f4a2]

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

lordk911 commented 2 years ago

is it because of connection pool ? I've config spark.kyuubi.engine.single.spark.session=true

yaooqinn commented 2 years ago

For kyuubi.engine.share.level=CONNECTION, num of connections = num of engines

lordk911 commented 2 years ago

For kyuubi.engine.share.level=CONNECTION, num of connections = num of engines

thanks