Open A-little-bit-of-data opened 1 month ago
why does your configuration start with trino.
?
BTW, if you touch kyuubi-defaults.conf
, you must restart the Kyuubi process. For your case, maybe you can modify spark-defaults.conf
then the newly invoked spark-submit
will see the updated Spark configurations.
The
spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL
parameter is configured, but after completion, the spark driver pod still exists and the status is completed. It needs to be manually deleted.
it was reported by other users too, but can not be reproducible on my side. it's likely there are some issues on K8s client construction, you may want to enable trace level logs for Kyuubi server process to find what happened on your K8s client construction
why does your configuration start with
trino.
?
It should be trino. Used to configure multiple tenants
BTW, if you touch
kyuubi-defaults.conf
, you must restart the Kyuubi process. For your case, maybe you can modifyspark-defaults.conf
then the newly invokedspark-submit
will see the updated Spark configurations.
I have deployed kyuubi on a container and controlled it through configmap. I only modified kyuubi-defaults.conf but not spark-defaults.conf. Do I need to restart kyuubi? In version 1.9.1, both kyuubi-defaults.conf and spark-defaults.conf are in configmap. Do I need to restart kyuubi to load the new configuration after modifying the configuration files?
oh, the underscore is part of markdown syntax, you need to quote your inline code by backtick
Kyuubi process consumes kyuubi-defaults.conf
and spark-submit
consumes spark-defaults.conf
, it depends on your expectation to update and restart the corresponding configuration files and processes.
Kyuubi process consumes
kyuubi-defaults.conf
andspark-submit
consumesspark-defaults.conf
, it depends on your expectation to update and restart the corresponding configuration files and processes.
Sorry, I don't quite understand what you mean. I understand that the spark on k8s task submitted by spark-submit is restarted after spark-defaults.conf is modified, but the new configuration is still not loaded. Under normal circumstances, I understand that I should have modified the spark-defaults.conf file under configmap kyuubi-spark, and the changes in the spark-defaults.conf file have been read in the kyuubi process. Then the spark on k8s task submitted by spark-submit again should load the spark-defaults.conf file I modified, instead of restarting the kyuubi application to read the modified content of the spark-defaults.conf file.
correct.
in short, the modified spark-defaults.conf
will take effect in the next spark-submit
, so if you want to change spark conf without restarting Kyuubi process, just modify spark-defaults.conf
correct.
in short, the modified
spark-defaults.conf
will take effect in the nextspark-submit
, so if you want to change spark conf without restarting Kyuubi process, just modifyspark-defaults.conf
But the current situation is that without restarting kyuubi, the content of spark-defaults.conf before the modification is always read. Only after restarting kyuubi can the modified content of spark-defaults.conf be read. Here, after modifying spark-defaults.conf, I checked the spark-defaults.conf file in the kyuubi pod. The spark-defaults.conf file in the pod has been updated, but using spark-submit still reads the old configuration of the spark-defaults.conf file
the content of spark-defaults.conf before the modification is always read.
this is not expected, how do you make this assertion, can you provide some logs/screenshots or other proofs?
the content of spark-defaults.conf before the modification is always read.
this is not expected, how do you make this assertion, can you provide some logs/screenshots or other proofs?
There is a time in the initial submission log - conf spark.kyuubi.engine.submit.time. That is, after changing the spark-defaults.conf file, I waited for a while and then restarted it. You can see that the startup configuration is still the same.
providing original logs.
Code of Conduct
Search before asking
Describe the bug
When kyuubi-defaults.conf is modified and spark is started again, the new configuration is not loaded
When kyuubi-defaults.conf is configured as follows,The spark task started on k8s still reads the old configuration.
trino.spark.executor.instances=2 trino.spark.driver.cores=1 trino.spark.executor.cores=2 trino.spark.kubernetes.driver.limit.cores=2 trino.spark.kubernetes.executor.limit.cores=3 trino.spark.driver.memory=1g trino.spark.executor.memory=2g
trino.spark.executor.instances=2 trino.spark.driver.cores=1 trino.spark.executor.cores=2 trino.spark.kubernetes.driver.limit.cores=2 trino.spark.kubernetes.executor.limit.cores=4 trino.spark.driver.memory=1g trino.spark.executor.memory=4g
The spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL parameter is configured, but after completion, the spark driver pod still exists and the status is completed. It needs to be manually deleted.
Affects Version(s)
1.9.1
Kyuubi Server Log Output
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?