Open AmitBenAmi opened 1 month ago
I found another use case in a different environment, with a version of v2.6.6
.
The behavior seems to be related to an app-of-apps
bootstrapping pattern, and it is always with the last application in the lexicographic order (literally the last one on the list).
This happened now with both ApplicationSet and an app-of-apps
with kustomize
with static Application manifests, which leads me to believe it isn't related to the ApplicationSet, but rather to some conflict and merge between 2 different unrelated applications within ArgoCD, and then the last application under the app-of-apps
is being set automatically with a different repoURL
, path
etc... than what is configured in our repo.
Might be related to https://github.com/argoproj-labs/argocd-image-updater/issues/896
Give a try to v2.12 or v2.13 as well.
Checklist:
argocd version
.Describe the bug
I have an ArgoCD ApplicationSet that uses
git
generators to generate applications. The design is:The
root parent app
usesgit
generator to find directories and based on them, create thesecond level parent app
. Thesecond level parent app
usesgit
generator to find*.yaml
files in that said directory and create applications based on theseyaml
files.After some time, for no apparent reason, one of the
second level parent app
is being deleted and causes issues on the cluster:namespaces
)second level parent app
hierarchy are fineThe main problem is that it leaves my cluster in a fatal state of not working (monitoring applications are deleted, essential workloads are deleted,
ArgoCD
itself is deleted because it manages itself, etc...).I suspect it takes over because the
app.kubernetes.io/instance
label on some namespaces that are terminating has the value of the application that was deleted.I didn't find a direct root cause or the specific issue, but I did see an error of that said app (before being deleted) that states a git message like:
That specific Commit Hash is from a different repo, that isn't being referenced in this application at all. While I see that message pops up occasionally, I do not see that application being deleted live, but only finds out about it when the cluster is in a fatal state already.
To Reproduce
Not sure how to reproduce, since it is a regular
ApplicationSet
withgit
generator for directories.Expected behavior
ArgoCD
(such as namespaces) aren't deletedScreenshots
Version
v2.6.6
Logs
Can't find the logs because the
ArgoCD
applicationset controller and all other pods were deleted, and unfortunately this cluster isn't set with persistent log archiving.I do have some other logs:
Edit:
Adding the log record that shows that this application is taking over all of the resources managed by
ArgoCD
Adding other record that shows that the
application-controller
updates theSyncStatus
with a different repo (the logs showsrepo-1
with its respective path, while the Application is configured withrepo-2
and its respective pathSo it makes me believe that my application is being replaced with other configurations somehow, and the repo and path are being changed for it