merge multiple files listed in KUBECONFIG environment var

adietish commented 2 months ago

Is your enhancement related to a problem? Please describe

One can provide kubectl with multiple kube config files via the KUBECONFIG env variable. Kubectl would merge those and let the user use them. ex.

export KUBECONFIG=${HOME}/.kube/config:${HOME}/.kube/minikube.yaml:${HOME}/.kube/sandbox.yaml

empty ~/.kube/config with only current-context pointing to a context that's defined in further file

apiVersion: v1
kind: Config
clusters: null
contexts: null
current-context: minikube
preferences: {}
users: null

~/.kube/minikube.yml

apiVersion: v1
kind: Config
clusters:
- cluster:
  certificate-authority: /Users/adietish/.minikube/ca.crt
  extensions:
    - extension:
        last-update: Wed, 07 Aug 2024 11:05:40 CEST
        provider: minikube.sigs.k8s.io
        version: v1.33.1
      name: cluster_info
  server: https://127.0.0.1:39093
name: minikube
contexts:
- context:
  cluster: minikube
  extensions:
    - extension:
        last-update: Wed, 07 Aug 2024 11:05:40 CEST
        provider: minikube.sigs.k8s.io
        version: v1.33.1
      name: context_info
  namespace: default
  user: minikube
name: minikube
users:
- name: minikube
user:
  client-certificate: /Users/adietish/.minikube/profiles/minikube/client.crt
  client-key: /Users/adietish/.minikube/profiles/minikube/client.key

~/.kube/sandbox.yml

apiVersion: v1
kind: Config
clusters:
- cluster:
  server: https://XXXX.openshiftapps.com:6443
name: api-sandbox-m3-1530-p1-openshiftapps-com:6443
users:
- name: adietish/XXXX-openshiftapps-com:6443
user:
  token: sha256~YYYYYYYY
contexts:
- context:
  cluster: XXXX-openshiftapps-com:6443
  namespace: adietish-dev
  user: adietish/XXXX-openshiftapps-com:6443
name: adietish-dev/XXXX-openshiftapps-com:6443/adietish

In kubectl I can then have config pointing to minikube or sandbox, both would work. In kubernetes-client, only the 1st file is used which then fails because it is missing url, user, etc.

Describe the solution you'd like

It would be great if kubernetes-client could merge these files into one config like kubectl does.

Describe alternatives you've considered

No response

Additional context

No response

adietish commented 2 months ago

I am working on a patch as this configuration/setup looks rather common when a user has lots of clusters. It allows a user to split up the configuration into smaller chunks rather to have a single big kube config.

adietish commented 2 months ago

Patch seems rather simple (just looking, didnt code so far): call io.fabric8.kubernetes.client.Config#mergeKubeConfigContents for each file that's present in the system variable, making sure that overrides/merges happen like in kubectl

adietish commented 2 months ago

For my own documentation: if the KUBECONFIG env variable is set (~/.kube/config:<additional file>:<additional file>), kubectl is using the first file in the list and adds

contexts
clusters
users

that are found in the additional files. The code for it is in client_go

rohanKanojia commented 2 months ago

if the KUBECONFIG env variable is set (~/.kube/config:<additional file>:<additional file>), kubectl is using the first file in the list and adds

@adietish : I'm sorry for replying late, after reading your comment I realized we had merged something similar a few years ago. However, I'm not sure whether it's completely aligned with kubectl behavior or not.

https://github.com/fabric8io/kubernetes-client/pull/1306

adietish commented 2 months ago

@rohanKanojia thx for the pointer, I had already looked into this. This PR only takes the 1st of the given files into account. All further ones are ignored. See here: https://github.com/fabric8io/kubernetes-client/pull/1306/files#diff-ee58b9555742a6e84ec59a74115410be46af05081afae5cf79a4516283ce37c7R444 There's a 2nd issue which add details to the logging warning but didnt' change the handling: #2174 (see https://github.com/fabric8io/kubernetes-client/pull/2181/files#diff-ee58b9555742a6e84ec59a74115410be46af05081afae5cf79a4516283ce37c7R477)

LOGGER.warn("Found multiple Kubernetes config files [{}], using the first one: [{}]. 
    If not desired file, please change it by doing `export KUBECONFIG=/path/to/kubeconfig` on Unix systems 
    or `$Env:KUBECONFIG=/path/to/kubeconfig` on Windows.", fileNames, fileNames[0]);

What I want to improve is to allow several files to be used. In https://github.com/quarkusio/quarkus/issues/8946#issuecomment-621144866 @manusa brought up how such a setup would work: Reading the kubectl docs, testing and reading of kubectl code suggests the following behaviour:

only the first file sets the current-context. None of the additional files in KUBECONFIG env var can set the current-context. (tested)
all the additional files listed in KUBECONFIG contribute contexts, clusters and users .

https://github.com/redhat-developer/intellij-kubernetes/issues/779 provides an exemplary setup where the 1st file sets the current-context but has no contexts, nor clusters, nor users. Then there are separate files for each cluster (ex. minikube.yml, sandbox.yml, etc.) which contribute a context, a cluster and a user.

manusa commented 2 months ago

Reading the kubectl docs, testing and reading of kubectl code suggests the following behaviour:

* only the first file sets the `current-context`. None of the additional files in `KUBECONFIG` env var can set the `current-context`.  (tested)

* all the additional files listed in `KUBECONFIG` contribute `contexts`, `clusters` and `users` .

This seems fine.

Implementation should be quite easy for merging. However, I think there are several places where the underlying configuration is updated (token refresh, etc.). These might be the tricky parts, since we'll probably need to traverse all configs until the config containing the token is found and updated.

adietish commented 2 months ago

@manusa thx for the pointer, I found https://github.com/fabric8io/kubernetes-client/blob/342fb8ac941ff10c8bedcb5824f9f2d33a86d341/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/utils/OpenIDConnectionUtils.java#L188-L225 Is this what you were referring to?

adietish commented 2 months ago

I have to specify my suggestion in https://github.com/fabric8io/kubernetes-client/issues/6240#issuecomment-2286276111 according to the Kubectl docs: kubeconfigs listed in the KUBECONFIG env var are fully merged while values in a latter file will not override a prior one. First in wins.

Never change the value or map key. Example: Preserve the context of the first file to set current-context. Example: If two files specify a red-user, use only values from the first file's red-user. Even if the second file has non-conflicting entries under red-user, discard them.

And then, when using kubectl --kubeconf <file> only a single file is allowed (mo merging happens).

If the --kubeconfig flag is set, use only the specified file. Do not merge. Only one instance of this flag is allowed.

fabric8io / kubernetes-client