fluxcd / flux

Successor: https://github.com/fluxcd/flux2
https://fluxcd.io
Apache License 2.0
6.89k stars 1.08k forks source link

flux runs kubectl apply even though there are no changes #1373

Closed wavemoran closed 6 years ago

wavemoran commented 6 years ago

flux will run kubectl apply -f every five minutes even though there are no changes to my images or git repo. I'm unsure if this is expected behavior, but this causes some chatty logs and makes alert mechanisms like fluxcloud useless.

Example log file:


ts=2018-09-18T17:00:39.503957407Z caller=sync.go:164 component=cluster method=Sync cmd="kubectl apply -f -" took=578.517456ms err=null output="namespace \"ingress\" configured\nnamespace \"monitoring\" configured\nnamespace \"srx\" configured\nnamespace \"weave\" configured\nservice \"admin-portal\" unchanged\nservice \"ambassador\" unchanged\nserviceaccount \"ambassador\" unchanged\nclusterrole \"ambassador\" configured\nservice \"ambassador-admin\" unchanged\nservice \"analytics\" unchanged\nservice \"client-portal\" unchanged\nserviceaccount \"fluentd\" unchanged\nclusterrole \"fluentd\" configured\nserviceaccount \"flux\" unchanged\nclusterrole \"flux\" configured\nclusterrole \"kiam-read\" configured\nservice \"kiam-server\" unchanged\nserviceaccount \"kiam-server\" unchanged\nclusterrole \"kiam-write\" configured\nclusterrole \"kube-cleaner-control\" configured\nserviceaccount \"kube-cleaner-service\" unchanged\nserviceaccount \"kube-state-metrics\" unchanged\nservice \"kube-state-metrics\" unchanged\nclusterrole \"kube-state-metrics\" configured\nrole \"kube-state-metrics-resizer\" unchanged\nservice \"member-portal\" unchanged\nservice \"memcached\" unchanged\nserviceaccount \"newrelic\" unchanged\nclusterrole \"newrelic\" configured\nserviceaccount \"sealed-secrets-controller\" unchanged\nservice \"sealed-secrets-controller\" unchanged\nrole \"sealed-secrets-key-admin\" unchanged\nclusterrole \"secrets-unsealer\" configured\nservice \"user-portal-api\" unchanged\nserviceaccount \"weave-net\" unchanged\nrole \"weave-net\" unchanged\nclusterrole \"weave-net\" configured\nclusterrolebinding \"admin-role-binding\" configured\nclusterrolebinding \"ambassador\" configured\nclusterrolebinding \"fluentd\" configured\nclusterrolebinding \"flux\" configured\nsecret \"flux-git-deploy\" unchanged\nclusterrolebinding \"kiam-read\" configured\nclusterrolebinding \"kiam-write\" configured\nclusterrolebinding \"kube-cleaner-service\" configured\nclusterrolebinding \"kube-state-metrics\" configured\nrolebinding \"kube-state-metrics\" unchanged\nclusterrolebinding \"newrelic\" configured\nclusterrolebinding \"sealed-secrets-controller\" configured\nrolebinding \"sealed-secrets-controller\" configured\nrolebinding \"srx-edit-binding\" configured\nconfigmap \"user-portal-config\" unchanged\nclusterrolebinding \"weave-net\" configured\nrolebinding \"weave-net\" unchanged\ndeployment \"admin-portal\" unchanged\ndeployment \"ambassador\" configured\ndeployment \"analytics-collector\" unchanged\ndeployment \"client-portal\" unchanged\ndaemonset \"fluentd-sumologic\" unchanged\ndeployment \"flux\" unchanged\ndaemonset \"kiam-agent\" unchanged\ndaemonset \"kiam-server\" unchanged\ncronjob \"kube-job-cleaner\" unchanged\ndeployment \"kube-state-metrics\" unchanged\ndeployment \"member-portal\" unchanged\ndeployment \"memcached\" unchanged\ndaemonset \"newrelic-infra\" unchanged\ndeployment \"sealed-secrets-controller\" unchanged\ndeployment \"user-portal-api\" unchanged\ndaemonset \"weave-net\" configured\ncustomresourcedefinition \"sealedsecrets.bitnami.com\" configured"
ts=2018-09-18T17:00:39.516739529Z caller=daemon.go:582 component=daemon event="Sync: 33bb60c..0cdad5f, default:clusterrole/ambassador, default:clusterrole/newrelic, default:clusterrole/secrets-unsealer, default:clusterrolebinding/admin-role-binding, default:clusterrolebinding/ambassador, default:clusterrolebinding/fluentd, default:clusterrolebinding/newrelic, default:clusterrolebinding/sealed-secrets-controller, default:customresourcedefinition/sealedsecrets.bitnami.com, default:namespace/monitoring, ingress:deployment/ambassador, ingress:service/ambassador-admin, ingress:serviceaccount/ambassador, kube-system:clusterrole/fluentd, kube-system:daemonset/fluentd-sumologic, kube-system:deployment/sealed-secrets-controller, kube-system:role/sealed-secrets-key-admin, kube-system:rolebinding/sealed-secrets-controller, kube-system:service/sealed-secrets-controller, kube-system:serviceaccount/fluentd, kube-system:serviceaccount/sealed-secrets-controller, monitoring:daemonset/newrelic-infra, monitoring:serviceaccount/newrelic, srx:configmap/user-portal-config, srx:deployment/admin-portal, srx:deployment/analytics-collector, srx:deployment/client-portal, srx:deployment/member-portal, srx:deployment/user-portal-api, srx:rolebinding/srx-edit-binding, srx:service/admin-portal, srx:service/analytics, srx:service/client-portal, srx:service/member-portal, srx:service/user-portal-api, weave:deployment/flux" logupstream=false
ts=2018-09-18T17:00:42.202641443Z caller=loop.go:399 component=sync-loop tag=flux-sync old=7fe661acf0038cbf31d8cc1af16e7e2f147129ae new=0cdad5feccc7ad234d77c81c45ad71398b652ea3
ts=2018-09-18T17:00:42.918505564Z caller=loop.go:100 component=sync-loop event=refreshed url=git@github.com:xxx/kubernetes branch=dev HEAD=0cdad5feccc7ad234d77c81c45ad71398b652ea3```
squaremo commented 6 years ago

flux will run kubectl apply -f every five minutes even though there are no changes to my images or git repo. I'm unsure if this is expected behavior, but this causes some chatty logs and makes alert mechanisms like fluxcloud useless.

Running every five minutes, whether or not there are changes in the git repo, is deliberate. We aren't in general party to changes made in the cluster, so to have any kind of reliable convergence, we need to maintain the state in git as well as effect changes. (If we were listening to all changes, we could make compensating changes to maintain the state -- but since kubectl apply effectively calculates those compensating changes for us, it's simpler to just apply everything).

Sending notifications on every sync is not expected. I can think of a couple of reasons it might happen:

  1. The git tag used as a high water mark is not pushed to the upstream git repo; that will cause fluxd to resend notifications (they are at-least-once). I think you've indicated in slack that this shouldn't be a problem, and it's not reported in the log above, either.

  2. fluxd genuinely thinks it's syncing new commits each time. We can only see one such occasion in the logs above; the tell-tale sign would be if it repeated the line naming the commits, with the same SHA1s.

ts=2018-09-18T17:00:42.202641443Z caller=loop.go:399 component=sync-loop tag=flux-sync old=7fe661acf0038cbf31d8cc1af16e7e2f147129ae new=0cdad5feccc7ad234d77c81c45ad71398b652ea3
wavemoran commented 6 years ago

Your second point does seem to be what is going on. Here's some sample entries from this morning:

ts=2018-09-19T13:57:48.414320844Z caller=loop.go:399 component=sync-loop tag=flux-sync old=5d5170cf7e2de39ec3869790935ddd620c7b005c new=92d93a797f99feb66fa647fef77da6168929fd06"
ts=2018-09-19T13:52:44.00238279Z caller=loop.go:399 component=sync-loop tag=flux-sync old=5d5170cf7e2de39ec3869790935ddd620c7b005c new=92d93a797f99feb66fa647fef77da6168929fd06"
ts=2018-09-19T13:47:40.100744941Z caller=loop.go:399 component=sync-loop tag=flux-sync old=5d5170cf7e2de39ec3869790935ddd620c7b005c new=92d93a797f99feb66fa647fef77da6168929fd06"

Entries calling out the "refreshed" head hash accompany the above entries:

ts=2018-09-19T13:57:31.255404378Z caller=loop.go:100 component=sync-loop event=refreshed url=git@github.com:SmithHealth/kubernetes branch=dev HEAD=92d93a797f99feb66fa647fef77da6168929fd06"
squaremo commented 6 years ago

Which commit does flux-sync point at in the upstream repo (that on github)?

wavemoran commented 6 years ago

Ah, the release pointer appears to be cycling between two branches. We have two fluxes pointing at a single repo with two branches (prod and dev). Perhaps that is causing the issue?

Each flux points at a different branch via the config file like below:

- --git-url=git@github.com:SmithHealth/kubernetes
- --git-branch=dev
squaremo commented 6 years ago

That'll be it! Give each one a different --git-label (which sets the tag and a ref for notes) and they'll stop fighting.

squaremo commented 6 years ago

I wonder how fluxd could do better to detect and warn about this situation.

wavemoran commented 6 years ago

Adding --git-label did the trick! There's one release for each branch now. Thank you for your help.