Open dnskr opened 6 months ago
@pan3793 @zwangsheng @chgl Could you please have a look at it? I would like to implement it (see point 5 with conf example), but it would be great to have your feedback and comments before.
Looks good to me!
maybe just one minor detail: make sure all changed configs contribute to the checksum annotation for the statefulset so kyuubi is automatically restarted: https://github.com/apache/kyuubi/blob/master/charts/kyuubi/templates/kyuubi-statefulset.yaml#L41
@chgl I'm afraid it is impossible to calculate and keep updated checksum for entities in sparkConf.from
list, i.e. for provided ConfigMaps
and Secrets
. We probably can use lookup function to find checksum value during chart release or update, but users can change those ConfigMaps
and Secrets
in between. Do you know any good (ideally standard) solution to solve the problem?
@dnskr , you're right of course! I hadn't considered the from
secrets/configmaps. In that case I believe https://github.com/stakater/Reloader is still the best way to go (would require making the annotations for the StatefulSet https://github.com/apache/kyuubi/blob/master/charts/kyuubi/templates/kyuubi-statefulset.yaml#L40 configurable, though).
quick question, is restarting kyuubi when spark default config updated a must? I believe spark config can be updated in runtime, once the configmap is updated, upcoming sessions will be affected
quick question, is restarting kyuubi when spark default config updated a must? I believe spark config can be updated in runtime, once the configmap is updated, upcoming sessions will be affected
@sudohainguyen Very good question. In the current implementation there is no Kyuubi auto restart on Spark config changes, but I assume it might be needed to stop running Spark immediately on some config changes.
yeah I think it depends on how users expect it it would be perfect if we can make it as a configurable option
Created draft implementation https://github.com/apache/kyuubi/pull/6521
Suggested approach and the draft implementation has an unsolved problem described below.
Some properties affect both helm chart and the application. For instance, .Values.server.rest.port
is used in Kubernetes resource definitions (Pod and Service ports) and as kyuubi.frontend.rest.bind.port
value in kyuubi-defaults.conf
file. So, kyuubi.frontend.rest.bind.port
value should be in sync with .Values.server.rest.port
property if the chart user provides custom (overrides default) kyuubi-defaults.conf
file.
Ideally, it would be great to move such properties outside the default kyuubi-defaults.conf
file and keep the file empty by default. I.e. the solution is to set kyuubi.frontend.rest.bind.port
in another way, for example using hardcoded environment variable or cli argument, but as far as I know, Kyuubi does not provide such capabilities at the momement.
I mean we cannot set kyuubi.frontend.rest.bind.port
as KYUUBI_FRONTEND_REST_BIND_PORT
env variable or cli argument --conf kyuubi.frontend.rest.bind.port= 10099
similar to Spark.
Any ideas?
@dnskr Thanks for continuously working on this area. Carry my idea from the Slack channel - overriding configurations via --conf
is preferred over env vars.
@dnskr Thanks for continuously working on this area. Carry my idea from the Slack channel - overriding configurations via
--conf
is preferred over env vars.
@pan3793 I created improvement proposal based on your suggestion: Provide Kyuubi configuration properties with cli options. I'll try to implement it, but also I'll be glad if someone takes the opportunity to implement it because I'm not sure I have enough context at the moment.
Code of Conduct
Search before asking
What would you like to be improved?
There is no clear common approach at the moment on how to configure Kyuubi deployment if the Helm chart is used. I would like to discuss requirements, limitations and different options to choose one approach to follow and support it in the Kyuubi Helm chart configuration. The problem has been mentioned and discussed in multiple issues and PRs, so the idea is to collect all opinions in one place and make the decision.
Configuration system of the Apache Kyuubi The configuration system allows to configure values using the following options (ordered from low to high prio):
Runtime options
JDBC Connection URLs
andSET Statements
Can be skipped in the discussion, because they used only when Kyuubi is up and running.Runtime option
Environment variables
Configured by{{ .Values.env }}
and{{ .Values.envFrom }}
value properties. The Helm chart users can specify environment variables to provide necessary configuration values with low effort if needed. The properties also allow to use provided (existing)ConfigMaps
andSecrets
as the sources of environment variables, for instance:Static options Represented by configuration files which should be located in each Kyuubi container in specific paths. In general case, the easiest way to provide files into Kubernetes pod (container) is to mount
ConfigMap
orSecret
to a specific path.[IN PROGRESS] Requirements Note: this section is in progress and subject to discuss.
ConfigMaps
under the hood from value properties ofvalue.yaml
file.Secrets
should never be created and managed by Helm chart because of security consideration!ConfigMaps
andSecrets
by resource name as a reference.ConfigMaps
andSecrets
with priority order. MultipleConfigMaps
andSecrets
might have key duplicates, so the implementation should clearly resolve the collision by merging keys in priority order.ConfigMaps
managed by the chart withConfigMaps
andSecrets
provided by user with priority order. The issue with key duplicates should be clearly resolved.ConfigMaps
managed by the chart should have the lowest prio.value.yaml
file. Some configuration files might be huge and complex, so the idea is to prevent identation issues invalues.yaml
file.ConfigMaps
andSecrets
from one or many configuration files. Users might have a lot of xml, properties and other files, so the idea is to help users to createConfigMap
andSecret
resources in a simple way.How should we improve?
[IN PROGRESS] Approach Note: this section is in progress and subject to discuss.
values.yaml
by system like Kyuubi, Hadoop, Spark, Trino etc.sparkConfDir: /opt/spark/conf sparkConf: ...
Use
from
property to specify list of existingConfigMaps
andSecrets
to be mounted to the configuration path of Kyuubi container.The implementation idea is to use Projected Volumes with
core/v1/SecretProjection
andcore/v1/ConfigMapProjection
entities. Also it will allow to mergeConfigMap
created fromfiles
property with the entities fromfrom
property.Move
xxxConfDir
property toxxxConf
property.Configuration example for Spark
Provide documentation with examples on how to set file content as a property when installing the chart, see Helm docs.
Provide documentation with examples on how to create
ConfigMap
from file or directory, see Kubernetes docs .Are you willing to submit PR?