search

rancher / rancher

Complete container management platform
http://rancher.com
Apache License 2.0
23.41k stars 2.97k forks source link

Panic in client-go/tools/cache/shared_informer.go:227 #15684

Open alena1108 opened 6 years ago

alena1108 commented 6 years ago

Following panic is observed on Drone builds:

E0918 17:41:36.915820   14412 runtime.go:66] Observed a panic: &errors.errorString{s:"unable to calculate an index entry for key \"local/m-9789d\" on index \"nodeAccess\": no preferred addresses found; known addresses: []"} (unable to calculate an index entry for key "local/m-9789d" on index "nodeAccess": no preferred addresses found; known addresses: [])
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/thread_safe_store.go:252
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/thread_safe_store.go:73
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/store.go:128
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/shared_informer.go:360
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/shared_informer.go:202
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:444
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/controller.go:150
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/controller.go:124
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/controller.go:124
/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/shared_informer.go:227
/usr/local/go/src/runtime/asm_amd64.s:1333

Might be imposed by us given unable to calculate an index entry for key, but shouldn't result in panic ideally

alena1108 commented 6 years ago

Based on client-go code, looks like the panic is intentional: https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/staging/src/k8s.io/client-go/tools/cache/thread_safe_store.go#L252

We have to look if there is anything that can be done on rancher side to prevent this from happening.