search

fluxcd / flux2

Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.
https://fluxcd.io
Apache License 2.0
6.36k stars 592 forks source link

Reconciliation stuck because of Sealed Secrets #4969

Open jmsariron opened 1 week ago

jmsariron commented 1 week ago

Describe the bug

We are having this issue, where, not sure how exactly, but sometimes the Flux reconciliation is failing because of Sealed Secrets, which are (As far as I know) correct.

Kustomization gets stuck in Reconciliation in Progress, getting the details I get a:

Warning  HealthCheckFailed  19m   kustomize-controller  health check failed after 59m30.097886897s: timeout waiting for: [SealedSecret/myapp/myapp-secrets status: 'InProgress']

Checking the kustomize-controllerpod logs, it shows a server-side apply completed with all elements being unchanged and then the Reconciation Failed log message, I'm kinda confused.

Going to the SealedSecret on the Namespace it shos as correctly applied and Synced, with the corresponding regular Secret generated.

This is happening only sometimes and we can't figure out why. Since I can't reproduce it 100% of time I'm looking for some help to debug this behaviour.

Steps to reproduce

We are using SealedSecrets on other projects and AFAIK the same way, sometimes it fails sometimes not, so I don't know how to reproduce it exactly

Expected behavior

I should reconcile just fine

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

N/A

Flux check

N/A

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

jmsariron commented 1 week ago

Adding a bit more info. for this case particular I:

  1. Deleted the Sealed Secret inside Kubernetes
  2. Deleted the kustomize-controller pod

When the new kustomize-controller started syncing everything, it fixed the kustomization and now appears as Synced with the Sealed Secret recreated from the git repo, so it seems to be something related to he controller?