Open killcity opened 6 years ago
Seeing the following panic across all three pods (replicaset). Not sure what is causing? Saw a similar bug with Prometheus that might help. https://github.com/prometheus/prometheus/issues/2178
E0424 15:56:57.232462 1 runtime.go:66] Observed a panic: &runtime.TypeAssertionError{interfaceString:"interface {}", concreteString:"cache.DeletedFinalStateUnknown", assertedString:"*v1.Pod", missingMethod:""} (interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:72 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:65 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/asm_amd64.s:509 /usr/local/go/src/runtime/panic.go:491 /usr/local/go/src/runtime/iface.go:172 /root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:253 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:187 <autogenerated>:1 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:257 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 goroutine 66 [running]: github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58 +0x111 panic(0x1397ca0, 0xc4208a8280) /usr/local/go/src/runtime/panic.go:491 +0x283 github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58 +0x111 panic(0x1397ca0, 0xc4208a8280) /usr/local/go/src/runtime/panic.go:491 +0x283 github.com/tczekajlo/kube-consul-register/controller/pods.(*Controller).Watch.func2(0x13e1220, 0xc420923600) /root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:253 +0x2ff github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(0x156f428, 0xc42027e030, 0xc42027e020, 0x13e1220, 0xc420923600) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:187 +0x49 github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc420574020, 0x13e1220, 0xc420923600) <autogenerated>:1 +0x62 github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.NewInformer.func1(0x13a7660, 0xc42043fe40, 0x13a7660, 0xc42043fe40) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:257 +0x38c github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4200100b0, 0xc42037a300, 0x0, 0x0, 0x0, 0x0) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420 +0x202 github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).processLoop(0xc42034a000) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x3c github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).(github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.processLoop)-fm() /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x2a github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc420478ec0) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc420561ec0, 0x3b9aca00, 0x0, 0x1303b01, 0xc42020c000) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xa1 github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc420478ec0, 0x3b9aca00, 0xc42020c000) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).Run(0xc42034a000, 0xc42020c000) /root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x1be github.com/tczekajlo/kube-consul-register/controller/pods.(*Controller).Watch(0xc420052340) /root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279 +0x2e1 main.main.func3(0x7f9cfbadf118, 0xc420052340) /root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141 +0x31 created by main.main /root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:140 +0x580
Seeing the following panic across all three pods (replicaset). Not sure what is causing? Saw a similar bug with Prometheus that might help. https://github.com/prometheus/prometheus/issues/2178