search

fluxcd / source-controller

The GitOps Toolkit source management component
https://fluxcd.io
Apache License 2.0
239 stars 188 forks source link

reconciliation for git source stuck with growing `workqueue_unfinished_work_seconds` #1432

Open ndemeshchenko opened 5 months ago

ndemeshchenko commented 5 months ago

Hi, On my flux installation in AWS EKS, version 

flux: v2.1.2
helm-controller: v0.36.2
image-automation-controller: v0.27.0
image-reflector-controller: v0.23.0
kustomize-controller: v1.1.1
notification-controller: v1.1.0
source-controller: v1.1.2

periodically I see source-controller being stuck and not completing/starting any reconciliation on git sources, while helm sources seems running fine.

I can't find anything in the source controller log during the time or before workqueue_unfinished_work_seconds starting to grow.

image

Any pointers on where to look to find a solution? Or what information might help to dig deeper

souleb commented 2 months ago

Do you see anything in the controller logs? Please use --log-level=debug. Please also check your consumption metrics. It would also help to have a git repository description.