github-actions[bot]
commented
1 year ago Hello @Swarvenstein, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.
Closed Swarvenstein closed 1 year ago
github-actions[bot]
commented
1 year ago Hello @Swarvenstein, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.
pan3793
commented
1 year ago Hi @Swarvenstein, I notice the engine log contains the following message
SparkTBinaryFrontendService: Received request of closing SessionHandle [0d8be737-c71d-4870-954b-831835390f09]
...
SparkSQLSessionManager: SessionHandle [0d8be737-c71d-4870-954b-831835390f09] is closed, current opening sessions 0
...
SparkSQLSessionManager: Session stopped due to shared level is Connection.It indicates that you are using the CONNECTION engine share level, which means every connection(session) will create a new spark application, and once the connection(session) is closed, the engine will self-terminate immediately to save resources.
If you want to share the engine(spark application) across sessions(connections), please set kyuubi.engine.share.level to another options like 'USER'
Ref: https://kyuubi.readthedocs.io/en/master/deployment/engine_share_level.html
pan3793
commented
1 year ago ... Why it's needed - driver start takes about 20-30s and we want to keep it 5-10 minutes in idle state to prevent this time overhead for kyuubi users in each separate request.
Sorry, I'm not a native engine speaker and am not very clear w/ your origin question, for my understanding, I guess you may worry about one of the following ones
the spark-submit takes a long period and kyuubi will treat the launch operation as failed when the launching time reaches the threshold. In this case, you can enlarge kyuubi.session.engine.initialize.timeout whose default value is PT3M
the spark-submit is a kind of heavy operation, and you want to keep the started engine alive for a period e.g(10min) even if it does not hold the active connection so that the following connection can reuse the engine. In this case, you can refer to my first reply to learn about "engine share level"
Swarvenstein
commented
1 year ago Hello @pan3793 It looks like your solution with changing the connection share level works and I'm really appreciate for that. Just to clarify all the aspect - there is no way to extend session idle timeout with CONNECTION engine share level?
pan3793
commented
1 year ago there is no way to extend session idle timeout with CONNECTION engine share level?
Yes, Kyuubi just silently ignores the kyuubi.session.engine.idle.timeout under CONNECTION share level. The reason is, under CONNECTION share level, the engine wouldn't be connected by new sessions, there is no reason to keep alive after the only session is closed.
Swarvenstein
commented
1 year ago Thanks again for your help with this issue. I think it can be closed.
Code of Conduct
Search before asking
Describe the bug
Hello. We are running kyuubi in k8s environment and faced an issue with adjusting driver idle timeout. By default it's going to Completed state after 50s of inactivity. We've tried to use the following parameters in kyuubi-configmap.yaml\kyuubi-defaults.conf:
None of them (we tried one by one and all of them together) changing the driver state changing timeout.
Why it's needed - driver start takes about 20-30s and we want to keep it 5-10 minutes in idle state to prevent this time overhead for kyuubi users in each separate request.
I will be very grateful if you have the opportunity and help to solve this issue.
Affects Version(s)
1.6.1
Kyuubi Server Log Output
Kyuubi Engine Log Output
Kyuubi Server Configurations
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?