hamelg
commented
10 months ago I get the same stacktrace with one of my argocd applications. Panic occurs when triggering a "Hard Refresh". Restarting the application controller does not fix the problem, the panic returns after restart. I only encounter this problem with a specific application that contains huge crd objects. The application controller panics only when this application is rendered with a cmp plugin. When not handled by a cmp plugin, the panic does not occur and "Hard refresh" works correctly.
Version
{
"Version": "v2.8.4+ebab8ec",
"BuildDate": "2023-10-30T19:24:59Z",
"GitCommit": "ebab8ec6259a6997fa3f310cddc539cb0c76b442",
"GitTreeState": "clean",
"GoVersion": "go1.20.10",
"Compiler": "gc",
"Platform": "linux/amd64",
"KustomizeVersion": "v5.1.0 unknown",
"HelmVersion": "v3.12.1+g1b5d1f0",
"KubectlVersion": "v0.24.2",
"JsonnetVersion": "v0.20.0",
"ExtraBuildInfo": "{Vendor Information: Red Hat OpenShift GitOps version: v1.10.1}"
}Logs
Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 236 [running]:
runtime/debug.Stack()
/usr/lib/golang/src/runtime/debug/stack.go:24 +0x65
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
/remote-source/argo_cd/app/controller/appcontroller.go:1337 +0x65
panic({0x341cb20, 0x6a0b2e0})
/usr/lib/golang/src/runtime/panic.go:890 +0x263
github.com/argoproj/gitops-engine/pkg/diff.structuredMergeDiff(0xc0131f3640)
/remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230607163028-425d65e07695/pkg/diff/diff.go:153 +0xe5
github.com/argoproj/gitops-engine/pkg/diff.StructuredMergeDiff(0xc0002e7ae0?, 0xc0131f3900?, 0x5?, {0x3a52a3b?, 0xc0135c2460?})
/remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230607163028-425d65e07695/pkg/diff/diff.go:133 +0x5a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc00c825d20, 0xc00c825d28, {0xc0131f3900, 0x5, 0x5})
/remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230607163028-425d65e07695/pkg/diff/diff.go:101 +0x1c8
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc006f19400, 0x41, 0xc000b22de4?}, {0xc006f19000, 0x41?, 0x0?}, {0xc0131f3900, 0x5, 0x5})
/remote-source/argo_cd/deps/gomod/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20230607163028-425d65e07695/pkg/diff/diff.go:646 +0x12b
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc009fb6d80?, 0xc000775900?, 0x3a7613f?}, {0xc006f18c00?, 0x3278840?, 0x7?}, {0x4c39318, 0xc0013a81b0?})
/remote-source/argo_cd/app/util/argo/diff/diff.go:271 +0x615
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0007c2b00, 0xc001919000, 0xc00168f440, {0xc001920190, 0x1, 0x1}, {0xc0007a1ce0?, 0x1, 0x1}, 0x1, ...)
/remote-source/argo_cd/app/controller/state.go:581 +0x36d2
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0002e8600)
/remote-source/argo_cd/app/controller/appcontroller.go:1444 +0x105e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
/remote-source/argo_cd/app/controller/appcontroller.go:740 +0x29
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x4bf9160, 0xc000ce0270}, 0x1, 0xc0006e9140)
/remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
/remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
/remote-source/argo_cd/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run
/remote-source/argo_cd/app/controller/appcontroller.go:739 +0x6be
jgwest
seth-acuitymd
andrii-korotkov-verkada
Checklist:
argocd version.Describe the bug
Attemped to sync a custom resource in one Application, before the CRD was created by another Application. This resulted in repeated nil pointer dereferences. The issue was resolved by terminating the affected application controller, and letting it restart.
To Reproduce
I have not attempted a reproduction, but putting a custom resource in an application, then syncing the CRD in another application after syncing the initial app may result in this
Expected behavior
No attempt to dereference nil pointer!
Version
Logs