search

kappnav / issues

kAppNav global issues
0 stars 0 forks source link

Unable to edit kappnav CR via the console (symptom: too many "reconciled messages") #135

Closed rlcundiff closed 4 years ago

rlcundiff commented 4 years ago

Kubernetes is continually driving the kappnav operator which is continually updating the kappnav CR. This prevents editing the kappnav CR via the console. Note that updating via 'kubectl apply` works ok.

Wed Feb 12 10:02:02 CST 2020
$ kubectl describe kappnav | grep "Resource Version"
  Resource Version:    8696416
$ kubectl describe kappnav | grep "Resource Version"
  Resource Version:    8696472
$ kubectl describe kappnav | grep "Resource Version"
  Resource Version:    8696501
$ date
Wed Feb 12 10:02:16 CST 2020

The operator log has this sequence repeatedly. The same four objects are always Status: updated.

{"level":"info","ts":1579734922.775042,"logger":"utils","msg":"Reconciled","Kind":"ServiceAccount","Name":"kappnav-sa","Status":"updated"}
{"level":"info","ts":1579734922.8253284,"logger":"utils","msg":"Reconciled","Kind":"Service","Name":"kappnav-ui-service","Status":"updated"}
{"level":"info","ts":1579734922.8425007,"logger":"utils","msg":"Reconciled","Kind":"Route","Name":"kappnav-ui-service","Status":"unchanged"}
{"level":"info","ts":1579734922.843359,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.deployment-liberty","Status":"unchanged"}
{"level":"info","ts":1579734922.8443322,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.deployment","Status":"unchanged"}
{"level":"info","ts":1579734922.844854,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.ingress","Status":"unchanged"}
{"level":"info","ts":1579734922.8453324,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.job","Status":"unchanged"}
{"level":"info","ts":1579734922.8458357,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.pod","Status":"unchanged"}
{"level":"info","ts":1579734922.8461657,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.route","Status":"unchanged"}
{"level":"info","ts":1579734922.8472967,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.actions.service","Status":"unchanged"}
{"level":"info","ts":1579734922.8476837,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.sections.application","Status":"unchanged"}
{"level":"info","ts":1579734922.8479178,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.sections.deployment","Status":"unchanged"}
{"level":"info","ts":1579734922.8481417,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.sections.service","Status":"unchanged"}
{"level":"info","ts":1579734922.8484485,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.status-mapping.deployment","Status":"unchanged"}
{"level":"info","ts":1579734922.849276,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.status-mapping.pod","Status":"unchanged"}
{"level":"info","ts":1579734922.8494856,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.status-mapping.service","Status":"unchanged"}
{"level":"info","ts":1579734922.8498287,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav.status-mapping-unregistered","Status":"unchanged"}
{"level":"info","ts":1579734922.8498883,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"builtin","Status":"unchanged"}
{"level":"info","ts":1579734922.8499224,"logger":"utils","msg":"Reconciled","Kind":"ConfigMap","Name":"kappnav-config","Status":"unchanged"}
{"level":"info","ts":1579734922.9462147,"logger":"utils","msg":"Reconciled","Kind":"Deployment","Name":"kappnav-ui","Status":"updated"}
{"level":"info","ts":1579734923.1424057,"logger":"utils","msg":"Reconciled","Kind":"Deployment","Name":"kappnav-controller","Status":"updated"}
mrglavas commented 4 years ago

@rlcundiff: I had spoken to the Appsody folks about this and they've had similar challenges with their operator in the past. Other controllers in Kubernetes may have a watch on some of the resources that our operator creates. That can end up in a loop where the operator and the other controller are thrashing, having different opinions on the "correct" state of the resource and that can lead to it being constantly updated.

ServiceAccount for instance has a list of pull secrets. Some additional ones might be getting injected that the operator is throwing out. We should check that we're playing fairly here and in the other places where updates are being triggered without changes to the CR.

arturdzm commented 4 years ago

This issue is caused by ServiceAccount kappnav-sa It always modifies imagePullSecrets by setting it to sa-kappnav But OpenShift will inject its own secret (required for internal registry) The data should be merged with a cluster not overwritten. This creates infinite reconcile loop. Exact lines to fix are here

https://github.com/kappnav/operator/blob/4d527803c86bc1176736f23afc92b297a4a9cb4c/pkg/utils/utils.go#L101

cvignola commented 4 years ago

See also https://github.com/kappnav/issues/issues/163

amylin1 commented 4 years ago

With the fixes, I don't see repeatedly ServiceAccount on "kappnav-sa" and Service on "kappnav-ui-service" reconciled updated messages. I can also update kappnav CR (i.e.logging info) via console successfully.