Metac constantly updating child objects

[adwski]

Question.

Metac is constantly updating child objects of each custom resource, disregarding resync interval. What could be done to stop it? With decent amount of CR objects it becomes very slow.

I could only find one possible cause about missing status subresource, but adding it did not solve the problem.

Version: quay.io/amitkumardas/metac:v0.4.0

K8s version: v1.16.15

CRD example:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2020-11-24T10:08:36Z"
  generation: 2
  name: myresource.group.cloud
  resourceVersion: "252506325"
  selfLink: /apis/apiextensions.k8s.io/v1/customresourcedefinitions/myresource.group.cloud
  uid: 94d4ca81-fed7-4725-a40c-ea807ffcd192
spec:
  conversion:
    strategy: None
  group: group.cloud
  names:
    kind: myresource
    listKind: myresourceList
    plural: myresource
    singular: myresource
  preserveUnknownFields: true
  scope: Namespaced
  versions:
  - name: v1alpha1
    served: true
    storage: true
    subresources:
      status: {}

[grzesuav]

After adding status subresource, did you deleted and recreated all CR's ?

[adwski]

After adding status subresource, did you deleted and recreated all CR's ?

I did not. But I restarted metac, thought It is stateless. Will try it, thanks.

[adwski]

After adding status subresource, did you deleted and recreated all CR's ?

I recreated all CR's with kubectl get .. -o yaml kubectl delete ... kubectl apply -f ...

But that didn't help, unfortunately.

[adwski]

Figured out the problem. For some reason I though that passing ResyncPeriodSeconds = 0 would result in infinite resync interval. But I was wrong. Changed my hook to pass it as null and constant updates stopped.

[adwski]

That wasn't so simple :)

Although there was mistakes from my side regarding parentLabels, there are still some issues I'd like to mention.

1. Ingress child resources have to be created by sync hook with "pathType": string("ImplementationSpecific"), otherwise k8s will add this field by itself, and that will trigger updates.

2. TargetPort has to be specified for service child resources, otherwise it will trigger updates as well.

3. For deployment child object, port list in pod template has to be empty list and not nil in case pod should not have any ports, otherwise it will trigger updates.

4. And finally in controller/common/manage_children.go, SetLastApplied() func in merge() sometimes makes DeepEqual() in updateChildren() to return false even if merged and observed objects are the same. It happens because lastApplied configuration may not be identical, even though the content is the same. I suppose it happens because sequence of env vars in pod template is different for each call. In my case it happened with deployment child object.

[grzesuav]

hi @adwski , some questions : ad 1 .

"pathType": string("ImplementationSpecific")

can you elaborate a bit, not sure what you have in mind. ad 2. Maybe for clarity can you pass here what look like Ingress resource initilay created and how looked like fixed version ? ad 3. can you add examples as well ? ad 4.

It happens because lastApplied configuration may not be identical, even though the content is the same. I suppose it happens because sequence of env vars in pod template is different for each call.

hmm it might be, but changing deepEqual will have significant implication on metacontroller behavior.

If it is not a problem can you put here examples ? I would like to check it, some of thing you mentioned can be caused by serialization, for some (4.) I guess it would be good to have mitigation, but maybe by excluding lastApplied rather than making generic change, without thorough testing

Cheers :)

[grzesuav]

also, for 4. I created https://github.com/metacontroller/metacontroller/issues/155 some time ago , but didn't started looking at this. In general sometimes the exact thing you want to change are labels / annotations, so when changing comparison method we need to be extra careful