search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.26k stars 398 forks source link

Allow to excluded annotations from translation #241

Closed mixitgit closed 2 years ago

mixitgit commented 2 years ago

We are running vcluster on Huawei cloud, so we are using huawei cloud provider. Unlike most of the cloud providers, to create loadbalancer it requires users to specify a bunch of annotations. To make user experience of our teams more smooth we are using mutating webhook, which adds this annotations automatically.

While using vcluster we faced the following problem: some annotations are getting overwritten during translation process When user creates a service of loadbalancer type it is annotated on the base cluster and some of this annotations are added via huwei cloud controller. And this annotations are not allowed to be changed: 

  Warning  SyncError               8m55s (x19169 over 10d)  service-syncer      Error syncing to physical cluster: services "hello-world-lb-x-default-x-test" is forbidden: can't modify service elb [kubernetes.io/elb.id] annotation
  Normal   EnsuringLoadBalancer    6s (x4015 over 13d)      service-controller  Ensuring load balancer
  Warning  SyncError               5s (x11 over 10s)        service-syncer      Error syncing to physical cluster: services "hello-world-lb-x-default-x-test" is forbidden: can't modify service elb [kubernetes.io/elb.class] annotation

(This error is produced by cloud controller) So I think it would be very useful to let users specify a set of annotations that should be ignored during translation I found that there is already support for a number of annotations in code

mixitgit commented 2 years ago

We are currently using this pr to solve our issue, by specifying annotations we want to exclude like this:

       - args:
          - ...
          - --exclude-annotations=kubernetes.io/elb.class,kubernetes.io/elb.mark,kubernetes.io/elb.subnet-id,kubernetes.io/session-affinity-mode

I think it might be also helpful for extending syncer with separate sustom controllers located on the physical cluster

FabianKramm commented 2 years ago

@mixitgit thanks for creating this issue! Let me understand this issue first as we already merge annotations and try to avoid overriding those that were not set within the vcluster itself: so the annotations are added in the host cluster and then overridden through existing annotations in the vcluster?

mixitgit commented 2 years ago

The annotations are first added in the host cluster and then they seem to be erased when the resource is synced back from the vcluster It seems like we got the problem on the previous version of vcluster, and #185 actually fixed this issue, I think it can be closed Tested on v0.4.5 and the problem doesn't seem to occure anymore

FabianKramm commented 2 years ago

@mixitgit okay great to know! I'll close this then