search

janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice
http://dvid.io
Other
197 stars 33 forks source link

Panic when using gbucket #300

Closed stuarteberg closed 5 years ago

stuarteberg commented 5 years ago

I'm seeing a panic when launching with the following TOML file:

/groups/flyem/data/scratchspace/dvid-configs/hemibrain-dvid-config.toml

As soon as NeuTu connects, a panic occurs. See log below.

The panic occurs with the latest commit (0.8.21.post30). The previous build (0.8.21) does not panic.

Panic log ``` Panic detected on request bergs-ws1/7U1dRJv6J3-000037: assignment to entry in nil map IP: 10.60.1.62:57230, URL: /api/node/a89e/grayscalejpeg_6/specificblocks Stack trace from localgrayscale: goroutine 286 [running]: github.com/janelia-flyem/dvid/dvid.ReportPanic(0xc4229e6e60, 0x9f, 0xc4200380b0, 0xe) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/dvid/utils.go:247 +0x85 github.com/janelia-flyem/dvid/server.recoverHandler.func1.1(0xc421d28120, 0x1b, 0xc423388800, 0x7f6cec16dd98, 0xc422a7eec0) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:744 +0x1f1 panic(0xda82e0, 0xfc8590) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/runtime/panic.go:502 +0x229 github.com/janelia-flyem/dvid/datatype/imageblk.(*Data).ServeHTTP(0xc4247121a0, 0xc42037a0a0, 0x20, 0xc4200a2378, 0xfd0b20, 0xc423365d40, 0xc423388800, 0x0) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/datatype/imageblk/imageblk.go:2136 +0x3d76 github.com/janelia-flyem/dvid/server.instanceSelector.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:1051 +0x5f8 net/http.HandlerFunc.ServeHTTP(0xc421cdc310, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.repoRawSelector.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:865 +0x38a net/http.HandlerFunc.ServeHTTP(0xc42000dcc0, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web.(*cStack).ServeHTTPC(0xc4246aaf60, 0xc423365c80, 0xc423365c50, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:50 +0x77 github.com/zenazn/goji/web.(*Mux).ServeHTTPC(0xc4233f9730, 0xc423365c80, 0xc423365c50, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/mux.go:53 +0x76 github.com/zenazn/goji/web.(*router).route(0xc4233f9308, 0xc4246aaf00, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/router.go:119 +0x7b github.com/zenazn/goji/web.(*mStack).newStack.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:88 +0x56 net/http.HandlerFunc.ServeHTTP(0xc42000dbe0, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.corsHandler.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:842 +0xb1 net/http.HandlerFunc.ServeHTTP(0xc42000dc00, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.recoverHandler.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:749 +0xbe net/http.HandlerFunc.ServeHTTP(0xc42000dc20, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.httpAvailHandler.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:729 +0x90 net/http.HandlerFunc.ServeHTTP(0xc42000dc40, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web/middleware.AutomaticOptions.func1(0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware/options.go:70 +0x8d net/http.HandlerFunc.ServeHTTP(0xc42000dc60, 0x7f6cec16dd98, 0xc422a7eec0, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web/middleware.Logger.func1(0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware/logger.go:35 +0x2b6 net/http.HandlerFunc.ServeHTTP(0xc42000dc80, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web.(*cStack).ServeHTTPC(0xc4246aaf00, 0xc423365c80, 0xc423365c50, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:50 +0x77 github.com/zenazn/goji/web.(*Mux).ServeHTTPC(0xc4233f92d0, 0xc423365c80, 0xc423365c50, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/mux.go:53 +0x76 github.com/zenazn/goji/web.(*router).route(0xc4202d95a8, 0xc4246aae70, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/router.go:119 +0x7b github.com/zenazn/goji/web.(*mStack).newStack.func1(0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:88 +0x56 net/http.HandlerFunc.ServeHTTP(0xc42000db80, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web/middleware.RequestID.func1(0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware/request_id.go:68 +0x20c net/http.HandlerFunc.ServeHTTP(0xc42000dba0, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web.(*cStack).ServeHTTP(0xc4246aae70, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:46 +0x68 github.com/zenazn/goji/web.(*Mux).ServeHTTP(0xc4202d9570, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/mux.go:45 +0x5e net/http.(*ServeMux).ServeHTTP(0x1497a80, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:2340 +0x130 net/http.serverHandler.ServeHTTP(0xc4233bfd40, 0xfd19a0, 0xc421d73340, 0xc423388800) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:2697 +0xbc net/http.(*conn).serve(0xc421df2320, 0xfd2760, 0xc4233f40c0) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1830 +0x651 created by net/http.(*Server).Serve /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:2798 +0x27b Panic detected on request bergs-ws1/7U1dRJv6J3-000038: assignment to entry in nil map IP: 10.60.1.62:57232, URL: /api/node/a89e/grayscalejpeg_7/specificblocks Stack trace from localgrayscale: goroutine 272 [running]: github.com/janelia-flyem/dvid/dvid.ReportPanic(0xc420016aa0, 0x9f, 0xc4200380b0, 0xe) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/dvid/utils.go:247 +0x85 github.com/janelia-flyem/dvid/server.recoverHandler.func1.1(0xc421d29020, 0x1b, 0xc421e44700, 0x7f6cec16dd98, 0xc422a7fc80) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:744 +0x1f1 panic(0xda82e0, 0xfc8590) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/runtime/panic.go:502 +0x229 github.com/janelia-flyem/dvid/datatype/imageblk.(*Data).ServeHTTP(0xc422a6c8f0, 0xc42037a0a0, 0x20, 0xc4200a25d0, 0xfd0b20, 0xc4246d3a40, 0xc421e44700, 0x0) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/datatype/imageblk/imageblk.go:2136 +0x3d76 github.com/janelia-flyem/dvid/server.instanceSelector.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:1051 +0x5f8 net/http.HandlerFunc.ServeHTTP(0xc421bfb460, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.repoRawSelector.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:865 +0x38a net/http.HandlerFunc.ServeHTTP(0xc4201633e0, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web.(*cStack).ServeHTTPC(0xc421bf8f90, 0xc4246d3980, 0xc4246d3950, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:50 +0x77 github.com/zenazn/goji/web.(*Mux).ServeHTTPC(0xc4233f9730, 0xc4246d3980, 0xc4246d3950, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/mux.go:53 +0x76 github.com/zenazn/goji/web.(*router).route(0xc4233f9308, 0xc420030270, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/router.go:119 +0x7b github.com/zenazn/goji/web.(*mStack).newStack.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:88 +0x56 net/http.HandlerFunc.ServeHTTP(0xc420010040, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.corsHandler.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:842 +0xb1 net/http.HandlerFunc.ServeHTTP(0xc420010060, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.recoverHandler.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:749 +0xbe net/http.HandlerFunc.ServeHTTP(0xc420010080, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/janelia-flyem/dvid/server.httpAvailHandler.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/janelia-flyem/dvid/server/web.go:729 +0x90 net/http.HandlerFunc.ServeHTTP(0xc4200100a0, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web/middleware.AutomaticOptions.func1(0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware/options.go:70 +0x8d net/http.HandlerFunc.ServeHTTP(0xc4200100c0, 0x7f6cec16dd98, 0xc422a7fc80, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web/middleware.Logger.func1(0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware/logger.go:35 +0x2b6 net/http.HandlerFunc.ServeHTTP(0xc420010100, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web.(*cStack).ServeHTTPC(0xc420030270, 0xc4246d3980, 0xc4246d3950, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:50 +0x77 github.com/zenazn/goji/web.(*Mux).ServeHTTPC(0xc4233f92d0, 0xc4246d3980, 0xc4246d3950, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/mux.go:53 +0x76 github.com/zenazn/goji/web.(*router).route(0xc4202d95a8, 0xc4200301e0, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/router.go:119 +0x7b github.com/zenazn/goji/web.(*mStack).newStack.func1(0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:88 +0x56 net/http.HandlerFunc.ServeHTTP(0xc420010000, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web/middleware.RequestID.func1(0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware/request_id.go:68 +0x20c net/http.HandlerFunc.ServeHTTP(0xc420010020, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1947 +0x44 github.com/zenazn/goji/web.(*cStack).ServeHTTP(0xc4200301e0, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/middleware.go:46 +0x68 github.com/zenazn/goji/web.(*Mux).ServeHTTP(0xc4202d9570, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/work/src/github.com/zenazn/goji/web/mux.go:45 +0x5e net/http.(*ServeMux).ServeHTTP(0x1497a80, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:2340 +0x130 net/http.serverHandler.ServeHTTP(0xc4233bfd40, 0xfd19a0, 0xc421d737a0, 0xc421e44700) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:2697 +0xbc net/http.(*conn).serve(0xc421da6e60, 0xfd2760, 0xc4200b1340) /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:1830 +0x651 created by net/http.(*Server).Serve /opt/conda/conda-bld/dvid_1544482507673/_build_env/go/src/net/http/server.go:2798 +0x27b ```