Closed bigdave777 closed 1 week ago
Same problem here EKS 1.29 Ambassador 3.9.1
NAME READY STATUS RESTARTS AGE
emissary-ingress-78b8fbc589-9h9g8 0/1 CrashLoopBackOff 4 (5s ago) 105s
emissary-ingress-7bdb565474-6r7f7 0/1 CrashLoopBackOff 3 (46s ago) 105s
emissary-ingress-7f74fcb567-ld9pz 0/1 CrashLoopBackOff 3 (42s ago) 104s
emissary-ingress-c9745cddd-hgxjw 0/1 CrashLoopBackOff 4 (4s ago) 104s
time="2024-04-16 15:54:29.4007" level=error msg="goroutine \"/watcher\" exited with error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:369\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nreflect.flag.mustBeAssignableSlow\n\t/usr/local/go/src/reflect/value.go:269\nreflect.flag.mustBeAssignable\n\t/usr/local/go/src/reflect/value.go:259\nreflect.Value.Set\n\t/usr/local/go/src/reflect/value.go:2254\nk8s.io/apimachinery/pkg/runtime.structFromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:556\nk8s.io/apimachinery/pkg/runtime.fromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:359\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructuredWithValidation\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:247\nk8s.io/apimachinery/pkg/runtime.(*unstructuredConverter).FromUnstructured\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/converter.go:274\nk8s.io/apimachinery/pkg/runtime.(*Scheme).unstructuredToTyped\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:539\nk8s.io/apimachinery/pkg/runtime.(*Scheme).convertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:453\nk8s.io/apimachinery/pkg/runtime.(*Scheme).ConvertToVersion\n\t/go/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:438\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.convertAnnotationObject\n\t/go/pkg/snapshot/v1/annotations.go:102\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.ValidateAndConvertObject\n\t/go/pkg/snapshot/v1/annotations.go:84\ngithub.com/emissary-ingress/emissary/v3/pkg/snapshot/v1.(*KubernetesSnapshot).PopulateAnnotations\n\t/go/pkg/snapshot/v1/annotations.go:46\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1.2\n\t/go/cmd/entrypoint/watcher.go:448\ngithub.com/emissary-ingress/emissary/v3/pkg/debug.(*Timer).Time\n\t/go/pkg/debug/timer.go:164\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate.func1\n\t/go/cmd/entrypoint/watcher.go:447\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.(*SnapshotHolder).K8sUpdate\n\t/go/cmd/entrypoint/watcher.go:544\ngithub.com/emissary-ingress/emissary/v3/cmd/entrypoint.watchAllTheThingsInternal.func2\n\t/go/cmd/entrypoint/watcher.go:278\ngithub.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1\n\t/go/vendor/github.com/datawire/dlib/dgroup/group.go:385\ngithub.com/datawire/dlib/derrgroup.(*Group).Go.func2\n\t/go/vendor/github.com/datawire/dlib/derrgroup/errgroup.go:132\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/watcher
[2024-04-16 15:54:29 +0000] [13] [INFO] Handling signal: int
[2024-04-16 15:54:29 +0000] [17] [INFO] Worker exiting (pid: 17)
[2024-04-16 15:54:29 +0000] [13] [INFO] Shutting down: Master
time="2024-04-16 15:54:29.8158" level=error msg="shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field" func=github.com/emissary-ingress/emissary/v3/pkg/busy.Main file="/go/pkg/busy/busy.go:87" CMD=entrypoint PID=1
FWIW I had an ambassador resource still running in my cluster, and once I had removed that this error stopped happening.
FWIW I had an ambassador resource still running in my cluster, and once I had removed that this error stopped happening.
Do you remember what's resource you removed?
FWIW I had an ambassador resource still running in my cluster, and once I had removed that this error stopped happening.
What resource did you deleted? thanks!
I believe it was a related to the auth service https://www.getambassador.io/docs/edge-stack/latest/topics/running/services/auth-service . In my particular case, it was nestled in the annotations for a service.
Closing as the issue appears resolved. Thanks @bigdave777 for sharing the details of your discovery.
Describe the bug Upon deployment of the Emisarry helm chart, all 3 pods end up crashing whilst throwing the following error
shut down with error error: PANIC: reflect: reflect.Value.Set using value obtained using unexported field
To Reproduce Deploy emissary using version: "8.9.1" and observe the logs I've added the following config
Expected behavior Emissary pods should remain in a healthy state following the charts deployment
Versions (please complete the following information):