search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.93k stars 428 forks source link

Load Balancer IP is not synced to host after editing `spec.loadBalancerIP` from vcluster #438

Open akbarkn opened 2 years ago

akbarkn commented 2 years ago

What happened?

I'm trying to create and edit a service from the cluster with LoadBalancer type. The host is using Calico and Metallb for the network. Here is the Metallb configmap:

apiVersion: v1
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.56.10-192.168.56.20
kind: ConfigMap
metadata:
  name: config
  namespace: metallb-system

Helm values:

{
  "api": {
    "image": "k8s.gcr.io/kube-apiserver:v1.23.1",
    "replicas": 1,
    "resources": {
      "limits": {
        "cpu": "500m",
        "memory": "512Mi"
      },
      "requests": {
        "cpu": "200m",
        "memory": "128Mi"
      }
    }
  },
  "etcd": {
    "image": "k8s.gcr.io/etcd:3.5.1-0",
    "storage": {
      "size": "5.0Gi"
    },
    "replicas": 1,
    "resources": {
      "limits": {
        "cpu": "500m",
        "memory": "512Mi"
      },
      "requests": {
        "cpu": "100m",
        "memory": "128Mi"
      }
    }
  },
  "sync": {
    "storageclasses": {
      "enabled": true
    },
    "networkpolicies": {
      "enabled": true
    },
    "priorityclasses": {
      "enabled": true
    },
    "serviceaccounts": {
      "enabled": true
    },
    "volumesnapshots": {
      "enabled": true
    },
    "persistentvolumes": {
      "enabled": true
    },
    "poddisruptionbudgets": {
      "enabled": true
    }
  },
  "syncer": {
    "replicas": 1,
    "resources": {
      "limits": {
        "cpu": "500m",
        "memory": "512Mi"
      },
      "requests": {
        "cpu": "100m",
        "memory": "128Mi"
      }
    }
  },
  "service": {
    "type": "ClusterIP"
  },
  "enableHA": false,
  "controller": {
    "image": "k8s.gcr.io/kube-controller-manager:v1.23.1",
    "replicas": 1,
    "resources": {
      "limits": {
        "cpu": "500m",
        "memory": "512Mi"
      },
      "requests": {
        "cpu": "200m",
        "memory": "128Mi"
      }
    }
  },
  "serviceCIDR": "10.1.0.0/16"
}

The service is successfully created and the ip is assigned at the first time. But, if I tried to edit the service and change the IP from vcluster, it's not getting synced with the host.

$ kubectl expose pod/test-pod --type LoadBalancer --port 80 --load-balancer-ip 192.168.56.10
service/test-pod exposed

Here I can see the service is created in the vcluster

$ kubectl --kubeconfig kubeconfig get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.1.169.191   <none>          443/TCP        1h
test-pod     LoadBalancer   10.1.139.169   192.168.56.10   80:31911/TCP   67s

and also host

$ kubectl -n k8s-7157e2ee-afd5-4658-9086-d00bc4f74eee get svc
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                  AGE
kube-dns-x-kube-system-x-mycluster   ClusterIP      10.1.204.189   <none>          53/UDP,53/TCP,9153/TCP   1h
mycluster                            ClusterIP      10.1.169.191   <none>          443/TCP                  1h
mycluster-api                        ClusterIP      10.1.244.33    <none>          443/TCP                  1h
mycluster-etcd                       ClusterIP      10.1.63.125    <none>          2379/TCP,2380/TCP        1h
mycluster-etcd-headless              ClusterIP      None           <none>          2379/TCP,2380/TCP        1h
mycluster-node-awid3                 ClusterIP      10.1.149.199   <none>          10250/TCP                1h
test-pod-x-default-x-mycluster       LoadBalancer   10.1.139.169   192.168.56.10  80:31911/TCP              2m53s

But, after editing spec.loadBalancerIP (using kubectl edit or kubectl apply or kubectl patch) from the vcluster to 192.168.56.11, The IP is not getting changed both in the cluster and host.

$ kubectl --kubeconfig kubeconfig get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.1.169.191   <none>          443/TCP        1h
test-pod     LoadBalancer   10.1.139.169   192.168.56.10   80:31911/TCP   5m7s

host:

$ kubectl -n k8s-7157e2ee-afd5-4658-9086-d00bc4f74eee get svc
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                  AGE
kube-dns-x-kube-system-x-mycluster   ClusterIP      10.1.204.189   <none>          53/UDP,53/TCP,9153/TCP   1h
mycluster                            ClusterIP      10.1.169.191   <none>          443/TCP                  1h
mycluster-api                        ClusterIP      10.1.244.33    <none>          443/TCP                  1h
mycluster-etcd                       ClusterIP      10.1.63.125    <none>          2379/TCP,2380/TCP        1h
mycluster-etcd-headless              ClusterIP      None           <none>          2379/TCP,2380/TCP        1h
mycluster-node-awid3                 ClusterIP      10.1.149.199   <none>          10250/TCP                1h
test-pod-x-default-x-mycluster       LoadBalancer   10.1.139.169   192.168.56.10  80:31911/TCP              7m5s

This is what I got from kubectl --kubeconfig kubeconfig get events

...
6m6s        Normal    IPAllocated        service/test-pod   Assigned IP ["192.168.56.10"]
6m6s        Normal    nodeAssigned       service/test-pod   announcing from node "awid"
19s         Normal    LoadbalancerIP     service/test-pod   192.168.56.10 -> 192.168.56.11
19s         Normal    LoadbalancerIP     service/test-pod   192.168.56.11 -> 192.168.56.10

If I change the value from the host, it works as expected:

$ kubectl --kubeconfig kubeconfig get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.1.169.191   <none>          443/TCP        1h
test-pod     LoadBalancer   10.1.139.169   192.168.56.11   80:31911/TCP   9m50s

host:

$ kubectl -n k8s-7157e2ee-afd5-4658-9086-d00bc4f74eee get svc
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                  AGE
kube-dns-x-kube-system-x-mycluster   ClusterIP      10.1.204.189   <none>          53/UDP,53/TCP,9153/TCP   1h
mycluster                            ClusterIP      10.1.169.191   <none>          443/TCP                  1h
mycluster-api                        ClusterIP      10.1.244.33    <none>          443/TCP                  1h
mycluster-etcd                       ClusterIP      10.1.63.125    <none>          2379/TCP,2380/TCP        1h
mycluster-etcd-headless              ClusterIP      None           <none>          2379/TCP,2380/TCP        1h
mycluster-node-awid3                 ClusterIP      10.1.149.199   <none>          10250/TCP                1h
test-pod-x-default-x-mycluster       LoadBalancer   10.1.139.169   192.168.56.11   80:31911/TCP             11m

What did you expect to happen?

The load balancer IP should be changed as specified if the value is edited from vcluster.

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

The host using Calico and Metallb

Host cluster Kubernetes version

``` $ kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"} ```

Host cluster Kubernetes distribution

``` kubeadm and k3s ```

vlcuster version

``` $ vcluster --version vcluster version 0.7.1 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

``` k8s ```

OS and Arch

``` OS: Linux Ubuntu 20.04.2 LTS Arch: x86 ```
FabianKramm commented 2 years ago

@akbarkn thanks for creating this issue! vcluster currently only handles the case that the host cluster is setting the load balancer ip, so setting it inside the vcluster only works on creation as values are copied to the host cluster for creation. But we might also want to allow setting it from inside the vcluster.

akbarkn commented 2 years ago

@FabianKramm Thanks for the answer. Nice to hear that you will handle the case in the future.

sandeep540 commented 2 years ago

@FabianKramm I am able to replicate the same, can you please assign this bug to work on!

matskiv commented 2 years ago

@sandeep540 Are you still working on this issue? If you got blocked and need help, please don't hesitate to reach out to us in vcluster channel in our Slack.

sandeep540 commented 2 years ago

Hi @matskiv , No I am currently not working on this! you can reassign it to someone