Open glasser opened 4 years ago
@glasser looking at this usecase now. I must have missed the notifications. I would make use of GenericController to handle this usecase. GenericController tries to manage a watch (in this case pod) with any arbitrary attachments (in this case service).
Note that GenericController is not available in this repo since this repo is no longer maintained. It currently available at https://github.com/AmitKumarDas/metac/ (a fork) before we get a proper home for this project.
I see. This really does work exactly like a DecoratorController if you're starting from a clean system (it is the standard DecoratorController demo, after all) — it's just that if you want to bootstrap yourself from a manual implementation to using metacontroller/metac, then you have to jump through some hoops.
We recently started using metacontroller. While we have several ideas of what we can do with it, the main motivation was to switch from some manually-created services per pod to something based on the service-per-pod example.
We found it a bit challenging to "take over" existing services. The whole point of doing a service per pod is to provide a consistent IP address behind a domain name. While we could have chosen new names for our services and migrated our configuration to use the new names after setting up SPP, we decided it would be nice to just find a way to reuse the existing service objects.
When metacontroller (or at least DecoratorController; I haven't tried CustomController at all) tries to make an attachment and fails because it already exists, metacontroller logs this in its logs. Note that it will not try again until the normal resync period is up (or you change the watched object manually). So even if we wanted to resolve this by decorating our statefulset and deleting the existing services and letting metacontroller re-create them (with new IPs, which isn't great), we would still have to come up with a way to get metacontroller to re-create faster. We came up with a hack where another
service-per-pod-resync
annotation, if set, got returned as the sync handler's reconnectAfterSeconds value, so we could switch a given statefulset from "resync at normal speed" to "resync once per second"; however, it would be nice to be able to separately configure the default resync period for "syncing after errors" vs "resync normally" so that we wouldn't have to do this.We discovered that to make metacontroller consider a DecoratorController to own an existing resource, we needed to add an ownerReferences entry and add a metacontroller annotation to it. We found that running patch commands like this helped:
Some notes:
~1
is the way to encode a slash in JSON patchesservice-per-pod
here is the name of our DecoratorControllerkubectl get -oyaml sts/zk
As a feature request: I think it would be nice if there was a built-in way for DecoratorControllers (at least) to take over existing resources. eg, there could be a field on the attachment list or the updatestrategy with a name like "takeOverExistingResource" that makes it also query for resources with no owner and no metacontroller annotation, and if it finds that it needs to "create" such a resource, it updates the existing one to add the annotation and ownerReferences instead of creating the resource. We would want this to be something you only set temporarily during the rollout (eg, perhaps it would log a warning at sync time if nothing was "reparented"). I don't know how hard this is to implement, as it would affect the set of watched objects pretty seriously... In any case, I hope the above is helpful to others!