Make it easier to take over existing resources

[glasser]

We recently started using metacontroller. While we have several ideas of what we can do with it, the main motivation was to switch from some manually-created services per pod to something based on the service-per-pod example.

We found it a bit challenging to "take over" existing services. The whole point of doing a service per pod is to provide a consistent IP address behind a domain name. While we could have chosen new names for our services and migrated our configuration to use the new names after setting up SPP, we decided it would be nice to just find a way to reuse the existing service objects.

When metacontroller (or at least DecoratorController; I haven't tried CustomController at all) tries to make an attachment and fails because it already exists, metacontroller logs this in its logs. Note that it will not try again until the normal resync period is up (or you change the watched object manually). So even if we wanted to resolve this by decorating our statefulset and deleting the existing services and letting metacontroller re-create them (with new IPs, which isn't great), we would still have to come up with a way to get metacontroller to re-create faster. We came up with a hack where another service-per-pod-resync annotation, if set, got returned as the sync handler's reconnectAfterSeconds value, so we could switch a given statefulset from "resync at normal speed" to "resync once per second"; however, it would be nice to be able to separately configure the default resync period for "syncing after errors" vs "resync normally" so that we wouldn't have to do this.

We discovered that to make metacontroller consider a DecoratorController to own an existing resource, we needed to add an ownerReferences entry and add a metacontroller annotation to it. We found that running patch commands like this helped:

kubectl -n "$NAMESPACE" patch service/zk-0 --type='json' -p '[{"op": "add", "path": "/metadata/annotations/metacontroller.k8s.io~1decorator-controller", "value": "service-per-pod"}, {"op": "add", "path": "/metadata/ownerReferences", "value": [{"apiVersion": "apps/v1", "blockOwnerDeletion": true, "controller": true, "kind": "StatefulSet", "name": "zk", "uid": "9ab1f9e9-368d-11e9-b6d0-42010a8001f7"}]}]'

Some notes:

• The ~1 is the way to encode a slash in JSON patches
• service-per-pod here is the name of our DecoratorController
• zk is the statefulset
• the ui was found in kubectl get -oyaml sts/zk
• This only worked well because the service spec that service-per-pod was creating was basically exactly the same as the one we had created manually
• metacontroller updates the existing resources in a standard merge/patch way, which meant that there were some extraneous things on our services that metacontroller wouldn't add (eg, some annotations and labels). After doing the "takeover" we could scale up and compare the existing services to the new ones to see what other changes were needed
• As mentioned above, we ran our hook with a low resync time while we were doing these patches so they would be picked up quickly (and tailed the metacontroller logs)
• we use updateStrategy InPlace on our DecoratorController. No idea how well this works with other approaches

As a feature request: I think it would be nice if there was a built-in way for DecoratorControllers (at least) to take over existing resources. eg, there could be a field on the attachment list or the updatestrategy with a name like "takeOverExistingResource" that makes it also query for resources with no owner and no metacontroller annotation, and if it finds that it needs to "create" such a resource, it updates the existing one to add the annotation and ownerReferences instead of creating the resource. We would want this to be something you only set temporarily during the rollout (eg, perhaps it would log a warning at sync time if nothing was "reparented"). I don't know how hard this is to implement, as it would affect the set of watched objects pretty seriously... In any case, I hope the above is helpful to others!

[AmitKumarDas]

@glasser looking at this usecase now. I must have missed the notifications. I would make use of GenericController to handle this usecase. GenericController tries to manage a watch (in this case pod) with any arbitrary attachments (in this case service).

Note that GenericController is not available in this repo since this repo is no longer maintained. It currently available at https://github.com/AmitKumarDas/metac/ (a fork) before we get a proper home for this project.

[glasser]

I see. This really does work exactly like a DecoratorController if you're starting from a clean system (it is the standard DecoratorController demo, after all) — it's just that if you want to bootstrap yourself from a manual implementation to using metacontroller/metac, then you have to jump through some hoops.