Closed wavemoran closed 6 years ago
flux will run
kubectl apply -f
every five minutes even though there are no changes to my images or git repo. I'm unsure if this is expected behavior, but this causes some chatty logs and makes alert mechanisms like fluxcloud useless.
Running every five minutes, whether or not there are changes in the git repo, is deliberate. We aren't in general party to changes made in the cluster, so to have any kind of reliable convergence, we need to maintain the state in git as well as effect changes. (If we were listening to all changes, we could make compensating changes to maintain the state -- but since kubectl apply
effectively calculates those compensating changes for us, it's simpler to just apply everything).
Sending notifications on every sync is not expected. I can think of a couple of reasons it might happen:
The git tag used as a high water mark is not pushed to the upstream git repo; that will cause fluxd to resend notifications (they are at-least-once). I think you've indicated in slack that this shouldn't be a problem, and it's not reported in the log above, either.
fluxd genuinely thinks it's syncing new commits each time. We can only see one such occasion in the logs above; the tell-tale sign would be if it repeated the line naming the commits, with the same SHA1s.
ts=2018-09-18T17:00:42.202641443Z caller=loop.go:399 component=sync-loop tag=flux-sync old=7fe661acf0038cbf31d8cc1af16e7e2f147129ae new=0cdad5feccc7ad234d77c81c45ad71398b652ea3
Your second point does seem to be what is going on. Here's some sample entries from this morning:
ts=2018-09-19T13:57:48.414320844Z caller=loop.go:399 component=sync-loop tag=flux-sync old=5d5170cf7e2de39ec3869790935ddd620c7b005c new=92d93a797f99feb66fa647fef77da6168929fd06"
ts=2018-09-19T13:52:44.00238279Z caller=loop.go:399 component=sync-loop tag=flux-sync old=5d5170cf7e2de39ec3869790935ddd620c7b005c new=92d93a797f99feb66fa647fef77da6168929fd06"
ts=2018-09-19T13:47:40.100744941Z caller=loop.go:399 component=sync-loop tag=flux-sync old=5d5170cf7e2de39ec3869790935ddd620c7b005c new=92d93a797f99feb66fa647fef77da6168929fd06"
Entries calling out the "refreshed" head hash accompany the above entries:
ts=2018-09-19T13:57:31.255404378Z caller=loop.go:100 component=sync-loop event=refreshed url=git@github.com:SmithHealth/kubernetes branch=dev HEAD=92d93a797f99feb66fa647fef77da6168929fd06"
Which commit does flux-sync
point at in the upstream repo (that on github)?
Ah, the release pointer appears to be cycling between two branches. We have two fluxes pointing at a single repo with two branches (prod and dev). Perhaps that is causing the issue?
Each flux points at a different branch via the config file like below:
- --git-url=git@github.com:SmithHealth/kubernetes
- --git-branch=dev
That'll be it! Give each one a different --git-label
(which sets the tag and a ref for notes) and they'll stop fighting.
I wonder how fluxd could do better to detect and warn about this situation.
Adding --git-label
did the trick! There's one release for each branch now. Thank you for your help.
flux will run
kubectl apply -f
every five minutes even though there are no changes to my images or git repo. I'm unsure if this is expected behavior, but this causes some chatty logs and makes alert mechanisms like fluxcloud useless.Example log file: