loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.92k stars 426 forks source link

Setting server in exportKubeConfig overwrites default secret #2234

Closed jhwagner closed 1 week ago

jhwagner commented 1 month ago

What happened?

When using exportKubeConfig to create an additional kubeconfig secret with a new server, it seems that both the new secret and the default (vc-NAME) secret get updated with the new server.

Unless I'm misunderstanding the docs, it seems like the exportKubeConfig configuration should only update the additional secret?

vCluster always creates a kubeconfig in a secret called vc-NAME in the namespace where you deployed vCluster. Configure exportKubeConfig so vCluster creates an additional secret with the given configuration.

from: https://www.vcluster.com/docs/vcluster/configure/vcluster-yaml/export-kube-config

What did you expect to happen?

I would expect the default vc-NAME kubeconfig secret to still be configured with the default localhost:8443 server.

How can we reproduce it (as minimally and precisely as possible)?

Create vcluster with following vcluster.yaml:

$ vcluster create cluster-a --namespace team-a --values vcluster.yaml
exportKubeConfig:
  server: https://cluster-a
  secret:
    name: vc-cluster-a-new

New kubeconfig secret (vc-cluster-a-new) is created as expected:

$ kubectl --namespace team-a get secret vc-cluster-a-new --template={{.data.config}} | base64 -d
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://cluster-a
  name: kubernetes
...

However the default secret's (vc-cluster-a) localhost:8443 seems to also have been overridden:

$ kubectl --namespace team-a get secret vc-cluster-a --template={{.data.config}} | base64 -d
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://cluster-a
  name: kubernetes
...

Anything else we need to know?

For context, I'm trying to have two versions of the kubeconfig, one with localhost that can be used to connect to the cluster via port forwarding and the other that can be used from within the host cluster for something like Flux to make API calls to the vcluster's apiserver. I could manually create the second secret, but would be nice of exportKubeConfig could be used.

Host cluster Kubernetes version

```console $ kubectl version Client Version: v1.31.1 Kustomize Version: v5.4.2 Server Version: v1.31.0 ```

vcluster version

```console $ vcluster --version vcluster version 0.20.1 ```

VCluster Config

``` exportKubeConfig: server: https://cluster-a secret: name: vc-cluster-a-new ```