search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.09k stars 913 forks source link

Session configs on first connection are applied by default to all other connections #6569

Open vontman opened 2 months ago

vontman commented 2 months ago

Code of Conduct

Search before asking

Describe the bug

How to reproduce

Using kyuubi + spark + hive

Initial Analysis

Affects Version(s)

1.7.1,1.9.2

Kyuubi Server Log Output

2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:java.library.path=/Users/omar.abdelbaset/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/folders/b3/xv1h8kxd3bq7y_hxpzwb1j600000gp/T/
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:os.name=Mac OS X
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:os.arch=aarch64
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:os.version=13.5.2
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:user.name=omar.abdelbaset
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:user.home=/Users/omar.abdelbaset
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Client environment:user.dir=/Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin
2024-07-29 14:57:28.804 INFO main org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=172.16.16.87:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@422b8438
2024-07-29 14:57:28.812 INFO main-SendThread(172.16.16.87:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket connection to server 172.16.16.87/172.16.16.87:2181. Will not attempt to authenticate using SASL (unknown error)
2024-07-29 14:57:28.812 INFO main org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is initialized.
2024-07-29 14:57:28.812 INFO main org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service[KyuubiTBinaryFrontend] is initialized.
2024-07-29 14:57:28.815 INFO main-SendThread(172.16.16.87:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection established to 172.16.16.87/172.16.16.87:2181, initiating session
2024-07-29 14:57:28.815 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.16.16.87:53641
2024-07-29 14:57:28.816 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is initialized.
2024-07-29 14:57:28.816 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is initialized.
2024-07-29 14:57:28.817 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /172.16.16.87:53641
2024-07-29 14:57:28.818 INFO main org.apache.kyuubi.server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is started.
2024-07-29 14:57:28.818 INFO main org.apache.kyuubi.server.PeriodicGCService: Service[PeriodicGCService] is started.
2024-07-29 14:57:28.819 INFO SyncThread:0 org.apache.kyuubi.shaded.zookeeper.server.persistence.FileTxnLog: Creating new log file: log.1
2024-07-29 14:57:28.819 INFO main org.eclipse.jetty.server.Server: jetty-9.4.54.v20240208; built: 2024-02-08T19:42:39.027Z; git: cef3fbd6d736a21e7d541a5db490381d95a2047d; jvm 1.8.0_332-b09
2024-07-29 14:57:28.824 INFO SyncThread:0 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Established session 0x1002a05dcbf0000 with negotiated timeout 60000 for client /172.16.16.87:53641
2024-07-29 14:57:28.824 INFO main-SendThread(172.16.16.87:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establishment complete on server 172.16.16.87/172.16.16.87:2181, sessionid = 0x1002a05dcbf0000, negotiated timeout = 60000
2024-07-29 14:57:28.828 INFO main-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2024-07-29 14:57:28.829 INFO Curator-ConnectionStateManager-0 org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Zookeeper client connection state changed to: CONNECTED
2024-07-29 14:57:28.840 INFO main org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@451882b2{/,null,AVAILABLE}
2024-07-29 14:57:28.845 INFO main org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@62f4ff3b{HTTP/1.1, (http/1.1)}{0.0.0.0:10019}
2024-07-29 14:57:28.846 INFO main org.eclipse.jetty.server.Server: Started @4700ms
2024-07-29 14:57:28.846 INFO main org.apache.kyuubi.metrics.PrometheusReporterService: Prometheus metrics HTTP server has started at http://172.16.16.87:10019/.
2024-07-29 14:57:28.846 INFO main org.apache.kyuubi.metrics.PrometheusReporterService: Service[PrometheusReporterService] is started.
2024-07-29 14:57:28.846 INFO main org.apache.kyuubi.metrics.MetricsSystem: Service[MetricsSystem] is started.
2024-07-29 14:57:28.848 INFO main org.apache.kyuubi.engine.KyuubiApplicationManager: Service[KyuubiApplicationManager] is started.
2024-07-29 14:57:28.848 INFO main org.apache.kyuubi.credentials.HadoopCredentialsManager: Service[HadoopCredentialsManager] is started.
2024-07-29 14:57:28.848 INFO main org.apache.kyuubi.server.metadata.MetadataManager: Service[MetadataManager] is started.
2024-07-29 14:57:28.849 INFO main org.apache.kyuubi.operation.KyuubiOperationManager: Service[KyuubiOperationManager] is started.
2024-07-29 14:57:28.849 INFO main org.apache.kyuubi.session.KyuubiSessionManager: Service[KyuubiSessionManager] is started.
2024-07-29 14:57:28.849 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiBackendService] is started.
2024-07-29 14:57:28.849 INFO KyuubiTBinaryFrontend: Thread-41 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Starting and exposing JDBC connection at: jdbc:hive2://localhost:10009/
2024-07-29 14:57:28.877 INFO main org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Created a /kyuubi/serverUri=localhost:10009;version=1.9.2;sequence=0000000000 on ZooKeeper for KyuubiServer uri: localhost:10009
2024-07-29 14:57:28.877 INFO main org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Registered KyuubiServiceDiscovery in namespace kyuubi.
2024-07-29 14:57:28.878 INFO main org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is started.
2024-07-29 14:57:28.878 INFO main org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service[KyuubiTBinaryFrontend] is started.
2024-07-29 14:57:28.878 INFO main org.eclipse.jetty.server.Server: jetty-9.4.54.v20240208; built: 2024-02-08T19:42:39.027Z; git: cef3fbd6d736a21e7d541a5db490381d95a2047d; jvm 1.8.0_332-b09
2024-07-29 14:57:28.878 INFO main org.eclipse.jetty.server.Server: Started @4733ms
2024-07-29 14:57:28.879 INFO main org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@466e91bb{HTTP/1.1, (http/1.1)}{localhost:10099}
2024-07-29 14:57:28.914 INFO main org.apache.kyuubi.server.http.authentication.AuthenticationFilter: Add authentication handler BasicAuthenticationHandler for scheme BASIC
2024-07-29 14:57:29.350 INFO main org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@10f19647{/api,null,AVAILABLE}
2024-07-29 14:57:29.500 INFO main org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@54336976{/engine-ui,null,AVAILABLE}
2024-07-29 14:57:29.500 INFO main org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5eabff6b{/,null,AVAILABLE}
2024-07-29 14:57:29.503 INFO main org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@664632e9{/ui,null,AVAILABLE}
2024-07-29 14:57:29.522 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is started.
2024-07-29 14:57:29.522 INFO main org.apache.kyuubi.server.KyuubiRestFrontendService: Exposing REST endpoint at: http://localhost:10099
2024-07-29 14:57:29.522 INFO main org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is started.
2024-07-29 14:57:29.526 INFO main org.apache.kyuubi.Utils: Loading Kyuubi properties from /Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin/conf/kyuubi-defaults.conf
2024-07-29 14:57:40.947 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
2024-07-29 14:57:40.951 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.session.KyuubiSessionManager: Opening session for anonymous@127.0.0.1
2024-07-29 14:57:40.966 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin/work/server_operation_logs/f4485708-0330-4761-b798-c08446d69c57/057fec04-a811-49a2-b444-b8870295cb99
2024-07-29 14:57:40.967 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[057fec04-a811-49a2-b444-b8870295cb99]: PENDING_STATE -> RUNNING_STATE, statement:
LaunchEngine
2024-07-29 14:57:40.968 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.session.KyuubiSessionManager: anonymous's KyuubiSessionImpl with SessionHandle [f4485708-0330-4761-b798-c08446d69c57] is opened, current opening sessions 1
2024-07-29 14:57:40.970 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2024-07-29 14:57:40.970 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=172.16.16.87:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@6f18edbd
2024-07-29 14:57:40.972 INFO KyuubiSessionManager-exec-pool: Thread-56-SendThread(172.16.16.87:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket connection to server 172.16.16.87/172.16.16.87:2181. Will not attempt to authenticate using SASL (unknown error)
2024-07-29 14:57:40.976 INFO KyuubiSessionManager-exec-pool: Thread-56-SendThread(172.16.16.87:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection established to 172.16.16.87/172.16.16.87:2181, initiating session
2024-07-29 14:57:40.976 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.16.16.87:53648
2024-07-29 14:57:40.976 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /172.16.16.87:53648
2024-07-29 14:57:40.977 INFO SyncThread:0 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Established session 0x1002a05dcbf0001 with negotiated timeout 60000 for client /172.16.16.87:53648
2024-07-29 14:57:40.977 INFO KyuubiSessionManager-exec-pool: Thread-56-SendThread(172.16.16.87:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establishment complete on server 172.16.16.87/172.16.16.87:2181, sessionid = 0x1002a05dcbf0001, negotiated timeout = 60000
2024-07-29 14:57:40.977 INFO KyuubiSessionManager-exec-pool: Thread-56-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2024-07-29 14:57:40.985 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1002a05dcbf0001 type:create cxid:0x2 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/kyuubi_1.9.2_SERVER_SPARK_SQL_lock/omar.abdelbaset/default/locks Error:KeeperErrorCode = NoNode for /kyuubi_1.9.2_SERVER_SPARK_SQL_lock/omar.abdelbaset/default/locks
2024-07-29 14:57:40.986 WARN KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.shaded.curator.utils.ZKPaths: The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.
2024-07-29 14:57:40.994 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1002a05dcbf0001 type:create cxid:0xd zxid:0xb txntype:-1 reqpath:n/a Error Path:/kyuubi_1.9.2_SERVER_SPARK_SQL_lock/omar.abdelbaset/default/leases Error:KeeperErrorCode = NoNode for /kyuubi_1.9.2_SERVER_SPARK_SQL_lock/omar.abdelbaset/default/leases
2024-07-29 14:57:41.021 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.engine.ProcBuilder: Logging to /Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin/work/omar.abdelbaset/kyuubi-spark-sql-engine.log.0
2024-07-29 14:57:41.024 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.engine.EngineRef: Launching engine:
/Users/user/Downloads/docker-v1.2-rtu/binaries/inx-downloads/spark-3.4.1--/spark-3.4.1--/bin/spark-submit \
    --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
    --conf spark.hive.merge.mapfiles=true; \
    --conf spark.hive.merge.mapredfiles=true; \
    --conf spark.hive.merge.size.per.task=128000000; \
    --conf spark.hive.merge.smallfiles.avgsize=128000000; \
    --conf spark.hive.merge.sparkfiles=true; \
    --conf spark.hive.merge.tezfiles=true; \
    --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
    --conf spark.kyuubi.client.ipAddress=172.16.16.87 \
    --conf spark.kyuubi.client.version=1.9.2 \
    --conf spark.kyuubi.engine.engineLog.path=/Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin/work/omar.abdelbaset/kyuubi-spark-sql-engine.log.0 \
    --conf spark.kyuubi.engine.share.level=SERVER \
    --conf spark.kyuubi.engine.submit.time=1722257861001 \
    --conf spark.kyuubi.ha.addresses=172.16.16.87:2181 \
    --conf spark.kyuubi.ha.engine.ref.id=f4485708-0330-4761-b798-c08446d69c57 \
    --conf spark.kyuubi.ha.namespace=/kyuubi_1.9.2_SERVER_SPARK_SQL/omar.abdelbaset/default \
    --conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
    --conf spark.kyuubi.server.ipAddress=127.0.0.1 \
    --conf spark.kyuubi.session.connection.url=localhost:10009 \
    --conf spark.kyuubi.session.real.user=anonymous \
    --conf spark.kyuubi.spark.metastore.db.ConnectionDriverName=com.mysql.jdbc.Driver \
    --conf spark.kyuubi.spark.metastore.db.ConnectionPassword=abc \
    --conf spark.kyuubi.spark.metastore.db.ConnectionURL=jdbc:mysql://localhost:23306/ms_db_docker_cluster_demo_9 \
    --conf spark.kyuubi.spark.metastore.db.ConnectionUserName=root \
    --conf spark.abc=123 \
    --conf spark.app.name=kyuubi_SERVER_SPARK_SQL_omar.abdelbaset_default_f4485708-0330-4761-b798-c08446d69c57 \
    --conf spark.cores.max=2 \
    --conf spark.driver.cores=1 \
    --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5009 \
    --conf spark.driver.memory=1g \
    --conf spark.eventLog.enabled=false \
    --conf spark.executor.cores=1 \
    --conf spark.executor.memory=1g \
    --conf spark.kubernetes.driver.label.kyuubi-unique-tag=f4485708-0330-4761-b798-c08446d69c57 \
    --conf spark.master=spark://localhost:27077 \
    --conf spark.port.maxRetries=100 \
    --conf spark.sql.adaptive.enabled=true \
    --conf spark.sql.ansi.enabled=true \
    --conf spark.sql.catalogImplementation=hive \
    --conf spark.yarn.tags=KYUUBI,f4485708-0330-4761-b798-c08446d69c57 \
    --proxy-user omar.abdelbaset /Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.2.jar
2024-07-29 14:57:56.696 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /172.16.16.87:53666
2024-07-29 14:57:56.698 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /172.16.16.87:53666
2024-07-29 14:57:56.699 INFO SyncThread:0 org.apache.kyuubi.shaded.zookeeper.server.ZooKeeperServer: Established session 0x1002a05dcbf0002 with negotiated timeout 60000 for client /172.16.16.87:53666
2024-07-29 14:57:56.731 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1002a05dcbf0002 type:create cxid:0x1 zxid:0x10 txntype:-1 reqpath:n/a Error Path:/kyuubi_1.9.2_SERVER_SPARK_SQL/omar.abdelbaset Error:KeeperErrorCode = NoNode for /kyuubi_1.9.2_SERVER_SPARK_SQL/omar.abdelbaset
2024-07-29 14:57:57.101 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Get service instance:172.16.16.87:53665 engine id:app-20240729125745-0006 and version:1.9.2 under /kyuubi_1.9.2_SERVER_SPARK_SQL/omar.abdelbaset/default
2024-07-29 14:57:57.270 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.session.KyuubiSessionImpl: [anonymous:172.16.16.87] SessionHandle [f4485708-0330-4761-b798-c08446d69c57] - Connected to engine [172.16.16.87:53665]/[app-20240729125745-0006] with SessionHandle [f4485708-0330-4761-b798-c08446d69c57]]
2024-07-29 14:57:57.271 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2024-07-29 14:57:57.274 INFO ProcessThread(sid:0 cport:2181): org.apache.kyuubi.shaded.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1002a05dcbf0001
2024-07-29 14:57:57.275 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x1002a05dcbf0001 closed
2024-07-29 14:57:57.275 INFO KyuubiSessionManager-exec-pool: Thread-56-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x1002a05dcbf0001
2024-07-29 14:57:57.275 INFO NIOServerCxn.Factory:/172.16.16.87:2181 org.apache.kyuubi.shaded.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.16.16.87:53648 which had sessionid 0x1002a05dcbf0001
2024-07-29 14:57:57.277 INFO KyuubiSessionManager-exec-pool: Thread-56 org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[057fec04-a811-49a2-b444-b8870295cb99]: RUNNING_STATE -> FINISHED_STATE, time taken: 16.308 seconds
2024-07-29 14:57:57.295 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.session.KyuubiSessionImpl: [anonymous:172.16.16.87] SessionHandle [f4485708-0330-4761-b798-c08446d69c57] - Starting to wait the launch engine operation finished
2024-07-29 14:57:57.295 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.session.KyuubiSessionImpl: [anonymous:172.16.16.87] SessionHandle [f4485708-0330-4761-b798-c08446d69c57] - Engine has been launched, elapsed time: 0 s
2024-07-29 14:58:12.030 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /Users/omar.abdelbaset/workplace/testing-kyuubi/apache-kyuubi-1.9.2-bin/work/server_operation_logs/f4485708-0330-4761-b798-c08446d69c57/c1fc9d85-69a8-4ef7-aab1-8902a9422077
2024-07-29 14:58:12.067 INFO KyuubiSessionManager-exec-pool: Thread-63 org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[c1fc9d85-69a8-4ef7-aab1-8902a9422077]: PENDING_STATE -> RUNNING_STATE, statement:
set spark.abc
2024-07-29 14:58:12.163 INFO KyuubiSessionManager-exec-pool: Thread-63 org.apache.kyuubi.operation.ExecuteStatement: Query[c1fc9d85-69a8-4ef7-aab1-8902a9422077] in FINISHED_STATE
2024-07-29 14:58:12.163 INFO KyuubiSessionManager-exec-pool: Thread-63 org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[c1fc9d85-69a8-4ef7-aab1-8902a9422077]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.095 seconds
2024-07-29 14:58:12.273 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.operation.ExecuteStatement: Processing anonymous's query[c1fc9d85-69a8-4ef7-aab1-8902a9422077]: FINISHED_STATE -> CLOSED_STATE, time taken: 0.205 seconds
2024-07-29 14:58:12.287 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-55 org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:C1 FC 9D 85 69 A8 4E F7 AA B1 89 02 A9 42 20 77, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

kyuubi.frontend.bind.host=localhost
kyuubi.engine.share.level=SERVER
spark.master=spark://localhost:7077
kyuubi.authentication.custom.class=org.apache.kyuubi.service.authentication.CustomAuthenticationProviderImpl
kyuubi.spark.metastore.db.ConnectionURL=jdbc:mysql://localhost:23306/ms_db
spark.sql.catalogImplementation=hive

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

github-actions[bot] commented 2 months ago

Hello @vontman, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.

pan3793 commented 2 months ago

this is by design, and is one of the known limitations of sharable spark engine. we provide some configurations to mitigate that in special cases, for example,

kyuubi.session.conf.ignore.list - A comma-separated list of ignored keys. If the client connection contains any of them, the key and the corresponding value will be removed silently during engine bootstrap and connection setup. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering but will not forbid users to set dynamic configurations via SET syntax.

kyuubi.session.conf.restrict.list - A comma-separated list of restricted keys. If the client connection contains any of them, the connection will be rejected explicitly during engine bootstrap and connection setup. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering but will not forbid users to set dynamic configurations via SET syntax.

vontman commented 2 months ago

@pan3793 Thanks for the quick reply!

May I ask why it is designed like this? Why we might need to append the session properties to the engine submission?

For the configuration kyuubi.session.conf.ignore.list, the ignored keys will not be set at all in the connection? or they will be ignored just for the bootstrap?

pan3793 commented 2 months ago

specific to spark engine, there is a concept "static configuration" which means it can only be set on spark app bootstrap, e.g. driver/executor cores/memory size, a typical use case is:

beeline -u "jdbc:kyuubi://kyuubi:10009/db1#kyuubi.engine.share.level=CONNECTION;spark.driver.memory=4g;spark.executor.memory=8g" -f batch-sql-1.sql
beeline -u "jdbc:kyuubi://kyuubi:10009/db2#kyuubi.engine.share.level=CONNECTION;spark.driver.memory=2g;spark.executor.memory=4g" -f batch-sql-2.sql

if kyuubi.session.conf.ignore.list is configured, the listed keys will be removed in both engine bootstrap and session context.

vontman commented 2 months ago

@pan3793 Many thanks for the information 🙏

I am afraid these configurations will not be of much help in our case 😅 But they are good to have for protection 👌

pan3793 commented 2 months ago

@vontman I removed the "bug" label as it is not, and you are always welcome to contribute ideas/patches to improve that :)