savar
commented
3 days ago I have a possible explanation for this (in case you also don't see any issues except the error message being logged).
-
The code in ArgoCD is trying (probably for performance reasons) first a
helm templatewithout thehelm dependency build: https://github.com/argoproj/argo-cd/blob/eca1f21d647ad7a099bb57ef73e7cb519eeb29e9/reposerver/repository/repository.go#L1201 and if the error is aIsMissingDependencyErr(which just matches for the error message from helm), then it will actually pull the dependencies and try it again.out, err := h.Template(templateOpts) if err != nil { if !helm.IsMissingDependencyErr(err) { return nil, err } if concurrencyAllowed { err = runHelmBuild(appPath, h) } else { err = h.DependencyBuild() } .... out, err = h.Template(templateOpts) if err != nil { return nil, err } } - the actual
h.Template()is runningout, err := h.cmd.template(".", templateOpts)(util/helm/helm.go) which is runningout, err := c.run(args...)(util/helm/cmd.go) which is callingreturn executil.RunWithRedactor(cmd, redactor)which inutil/exec/exec.gois then usingopts := ExecRunOpts{Redactor: redactor}whileExecRunOptsis a struct which has also a bool forSkipErrorLoggingwhich per default becomesfalseand this is then finally callingreturn argoexec.RunCommandExt(cmd, cmdOpts)which is in another package - the other package will in https://github.com/argoproj/pkg/blob/58a04f832405dd3e6091bfea848a25b5764817c5/exec/exec.go#L157 then log the error
if !opts.SkipErrorLogging { logCtx.Error(err.Error()) }so that we see the error in the logfile. A fix would be, to set SkipErrorLogging to
trueas it is actually just annoying "noise". But this needs to be verified and checked that the error is indeed logged on the ArgoCD part if it is a real error, so that we do not lose the error message when it has to be logged. But I didn't check that fully, so this has to be done by someone else (or I will have to check if I can create a Pull Request, but not sure if I can manage that in a timely manner).
Also to trigger this issue I was able to do that by going to the UI on an application (helm) with a dependency in the Charts.yaml and doing a Hard Refresh (the small dropdown button next to Refresh).
Checklist:
argocd version.Describe the bug
We run an ArgoCD instance with some 300 applications and are regularly seeing the following error on different helm applications.
What I've noticed is that whenever this error occurs then the ordering of operations seems incorrect for example.
helm template< This should be lasthelm repo add Xhelm dependency buildNote: It happens on all different git repositories as well as helm repositories so I don't believe it's unique to any particular setup, just happens more commonly with higher number of applications.
I assume that it's to do with this code running in parallel in another thread which causes the operations to be in the incorrect order.
Possibly related issues: https://github.com/argoproj/argo-cd/issues/11564 https://github.com/argoproj/argo-cd/issues/11564
To Reproduce
It's quite difficult for this to be replicated as I believe it to be due to a timing issue, but a suggested way might be to do the following.
For us it triggers a few hundred times an hour with out 300+ applications.
Expected behavior
helm templateshould never run beforehelm dependency build1Screenshots
N/A
Version
Logs
Easier to read Log
Full Log