search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.15k stars 368 forks source link

Random IPFamily in IPv6 Host Cluster #1898

Open thomas-bottesch opened 2 days ago

thomas-bottesch commented 2 days ago

What happened?

I have an IpV6 host cluster. However, when the vcluster control plane is restarted then it randomly chooses either IPv4 or IPv6 as its underlying cluster family. This causes various issues.

To show what is happening: I retrieved the command lines from the control plane pod after restarts.

/binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1
/binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1

Here we can see that the service-cluster-ip-range=10.96.0.0/12 which makes the cluster be an IPv4 cluster.

After restarting the pod now it gets

/binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=fd22:663c:50ba::/108 --bind-address=127.0.0.1
/binaries/kube-controller-manager --service-cluster-ip-range=fd22:663c:50ba::/108 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1

Where service-cluster-ip-range=fd22:663c:50ba::/108 which makes the cluster be in the IPv6 mode.

What did you expect to happen?

I would have expected that the vcluster would always have the IPv6 family (since host cluster is IPv6) or at least not change the family after a restart.

How can we reproduce it (as minimally and precisely as possible)?

I guess someone would have to create an IPv6 cluster and restart the vcluster control plane a few times to reproduce the issue.

Anything else we need to know?

Workaround:

To the top level of the vcluster.yaml one can set an undocumented value:

serviceCIDR: "fd22:663c:50ba::/108" to force the vcluster to be of IpV6 family serviceCIDR: "10.96.0.0/12" to force the vcluster to be of IpV4 family

Host cluster Kubernetes version

``` Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.10", GitCommit:"5c1d2d4295f9b4eb12bfbf6429fdf989f2ca8a02", GitTreeState:"clean", BuildDate:"2023-01-18T19:15:31Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"29+", GitVersion:"v1.29.4-eks-036c24b", GitCommit:"9c0e57823b31865d0ee095997d9e7e721ffdc77f", GitTreeState:"clean", BuildDate:"2024-04-30T23:53:58Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"} ```

vcluster version

``` vcluster version 0.20.0-beta.5 ```

VCluster Config

``` controlPlane: backingStore: etcd: deploy: enabled: true statefulSet: persistence: volumeClaim: storageClass: "vcluster-volume-sc" distro: eks: enabled: true proxy: extraSANs: - test3.some-url.net ingress: enabled: true host: test3.some-url.net spec: tls: - hosts: - test3.some-url.net ingressClassName: nginx annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: "true" exportKubeConfig: server: https://test3.some-url.net sync: toHost: serviceAccounts: enabled: true storageClasses: enabled: true # Networking options related to the virtual cluster. networking: advanced: # FallbackHostCluster allows to fallback dns to the host cluster. This is useful if you want to reach host services without # any other modification. You will need to provide a namespace for the service, e.g. my-other-service.my-other-namespace fallbackHostCluster: true ```
FabianKramm commented 1 day ago

@thomas-bottesch thanks for creating this issue! We will take a look shortly