jonescobedo
commented
3 years ago Same here although I created a dummy certificate within Azure Keyvault via Generate/Import, I increased the log level to "Trace" see below
{"ts":1616649528797.6128,"msg":"started workers","v":0}
{"ts":1616649528797.7607,"msg":"processing azurekeyvaultsecret","v":4,"key":"vnext-backend/svrs-azr-dev-vital-test-cert"}
{"ts":1616649528797.7974,"msg":"get or create secret","v":4,"secret":"vnext-backend/nginx-cert"}
{"ts":1616649529050.7837,"msg":"values have changed requiring update to secret","v":0,"azurekeyvaultsecret":"vnext-backend/svrs-azr-dev-vital-test-cert","secret":"vnext-backend/nginx-cert"}
{"ts":1616649529051.0198,"msg":"Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 83 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic(0x184ca20, 0x2713ba0)\n\t/go
/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:74 +0x95\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:48 +0x89\npanic(0x184ca20, 0x2713ba0)\n\t/usr/local/go/src/ru
ntime/panic.go:969 +0x1b9\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller.createNewSecretFromExisting(0xc000435180, 0xc0002985a0, 0xc0006bec80, 0x0, 0x0, 0x1a87cc2)\n\t/go/src/github.com/SparebankenVest/azure-key-vault-t
o-kubernetes/cmd/azure-keyvault-controller/controller/secret.go:227 +0x6ef\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller.(*Controller).getOrCreateKubernetesSecret(0xc0004269c0, 0xc000435180, 0x2a, 0xc000435180, 0x0)\n\
t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller/secret.go:162 +0x7a9\ngithub.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller.(*Controller).syncAzureKeyVaultSecret(0xc000
4269c0, 0xc0000a37d0, 0x2a, 0xc00046f4e0, 0x7277575479596a00)\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller/azureKeyVaultSecret.go:168 +0x67b\nkmodules.xyz/client-go/tools/queue.(*Worker).processNextEntry(0xc
0002cca00, 0x203000)\n\t/go/pkg/mod/kmodules.xyz/client-go@v0.0.0-20200521013203-6fe0a448d053/tools/queue/worker.go:84 +0xec\nkmodules.xyz/client-go/tools/queue.(*Worker).processQueue(0xc0002cca00)\n\t/go/pkg/mod/kmodules.xyz/client-go@v0.0.0-20200521013203-6fe0a448d053/
tools/queue/worker.go:67 +0x2b\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0003f6030)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 +0x5f\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0003f6030, 0x1c4d8a0, 0xc0002984b0, 0x6a684465
76386901, 0xc000082120)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 +0xad\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003f6030, 0x3b9aca00, 0x0, 0x6b647a4c31426e01, 0xc000082120)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wai
t.go:133 +0x98\nk8s.io/apimachinery/pkg/util/wait.Until(0xc0003f6030, 0x3b9aca00, 0xc000082120)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90 +0x4d\ncreated by kmodules.xyz/client-go/tools/queue.(*Worker).Run\n\t/go/pkg/mod/kmodules.xyz/client-go@v0
.0.0-20200521013203-6fe0a448d053/tools/queue/worker.go:53 +0x89\n","v":0}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x16d9a6f]
goroutine 83 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x184ca20, 0x2713ba0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller.createNewSecretFromExisting(0xc000435180, 0xc0002985a0, 0xc0006bec80, 0x0, 0x0, 0x1a87cc2)
/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller/secret.go:227 +0x6ef
github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller.(*Controller).getOrCreateKubernetesSecret(0xc0004269c0, 0xc000435180, 0x2a, 0xc000435180, 0x0)
/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller/secret.go:162 +0x7a9
github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller.(*Controller).syncAzureKeyVaultSecret(0xc0004269c0, 0xc0000a37d0, 0x2a, 0xc00046f4e0, 0x7277575479596a00)
/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-controller/controller/azureKeyVaultSecret.go:168 +0x67b
kmodules.xyz/client-go/tools/queue.(*Worker).processNextEntry(0xc0002cca00, 0x203000)
/go/pkg/mod/kmodules.xyz/client-go@v0.0.0-20200521013203-6fe0a448d053/tools/queue/worker.go:84 +0xec
kmodules.xyz/client-go/tools/queue.(*Worker).processQueue(0xc0002cca00)
/go/pkg/mod/kmodules.xyz/client-go@v0.0.0-20200521013203-6fe0a448d053/tools/queue/worker.go:67 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0003f6030)
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0003f6030, 0x1c4d8a0, 0xc0002984b0, 0x6a68446576386901, 0xc000082120)
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003f6030, 0x3b9aca00, 0x0, 0x6b647a4c31426e01, 0xc000082120)
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0003f6030, 0x3b9aca00, 0xc000082120)
/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90 +0x4d
created by kmodules.xyz/client-go/tools/queue.(*Worker).Run
/go/pkg/mod/kmodules.xyz/client-go@v0.0.0-20200521013203-6fe0a448d053/tools/queue/worker.go:53 +0x89
181192
jcristol
TomasTokaMrazek
Note: Make sure to check out known issues (https://akv2k8s.io/troubleshooting/known-issues/) before submitting
Components and versions Select which component(s) the bug relates to with [X].
[X ] Controller, version:
x.x.x(docker image tag) [ ] Env-Injector (webhook), version:x.x.x(docker image tag) [ ] OtherDescribe the bug A clear and concise description of what the bug is. When syncing a certificate following this tutorial the controller crashes while trying to get the certificate for azure.
To Reproduce Steps to reproduce the behavior: Try syncing a pem certificate signed by digicert from an azure keyvault.
Expected behavior A clear and concise description of what you expected to happen. I would expect the controller not to crash
Logs If applicable, add logs to help explain your problem.
Crash output
Additional context Add any other context about the problem here.
We noticed this happening on certificates that were previously getting synced. The last sync date for the certificate causing the crash was
2021-03-02.We haven't updated the certificate in the keyvault nor had we touched our azure key vault controller config so I suspect its a bug or a change to Azure's api.