runtime.errorString: runtime error: invalid memory address or nil pointer dereference

openshift-sentry commented 7 years ago

https://sentry.io/red-hat/openshift-3-dev-preview-int/issues/333820561/

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 514, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 489, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 63, in panicmem
...
(8 additional frame(s) were not displayed)

runtime error: invalid memory address or nil pointer dereference

smarterclayton commented 7 years ago

@sjenning @derekwaynecarr just started happening.

sjenning commented 7 years ago

@smarterclayton how does one get access to sentry?

smarterclayton commented 7 years ago

runtime.errorString: runtime error: invalid memory address or nil pointer dereference
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/pkg/cmd/util/serviceability/panic.go", line 28, in CapturePanic-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go", line 51, in HandleCrash
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 514, in call32
  File "/usr/lib/golang/src/runtime/panic.go", line 489, in gopanic
  File "/usr/lib/golang/src/runtime/panic.go", line 63, in panicmem
  File "/usr/lib/golang/src/runtime/signal_unix.go", line 290, in sigpanic
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/pkg/sdn/plugin/proxy.go", line 298, in OnEndpointsUpdate
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/proxy/config/config.go", line 142, in handleUpdateEndpoints
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/proxy/config/config.go", line 84, in handleUpdateEndpoints)-fm
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go", line 199, in OnUpdate
  File "<autogenerated>", line 58, in OnUpdate
  File "/builddir/build/BUILD/atomic-openshift-git-0.c420cf9/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go", line 545, in run
  File "/usr/lib/golang/src/runtime/asm_amd64.s", line 2197, in goexit

Commit is in openshift/ose@c420cf9.

sjenning commented 7 years ago

    pep := proxy.allEndpoints[ep.UID]
    if pep == nil {
        glog.Warningf("Got OnEndpointsUpdate for unknown Endpoints %#v", ep)
        pep := &proxyEndpoints{ep, true}  <- bad scope
        proxy.allEndpoints[ep.UID] = pep
    }
    wasBlocked := pep.blocked <- crash here

Possible the endpoints changes are causing us to hit this case where we weren't before

danwinship commented 7 years ago

Possible the endpoints changes are causing us to hit this case where we weren't before

What endpoints changes?

sjenning commented 7 years ago

@danwinship not sure exactly. @smarterclayton has been alluding to some changes made to mitigate endpoint update rates that are not this PR https://github.com/openshift/origin/pull/15888

openshift / origin

runtime.errorString: runtime error: invalid memory address or nil pointer dereference #15922