panic in pool.go:279 - Githubissues

go-libipfs/gateway code recovers from panics and logs them. below one was found in bifrost-gateway.

iiuc panic originates from caboose@v0.0.0-20230216184950-d2749d3b8402/pool.go:279 (bifrost-gateway uses latest code from https://github.com/filecoin-saturn/caboose/pull/26)

pool.go:279 panics due to index out of range [997] with length 997

observed it happen at bifrost-bank1-ny at least twice, with the same error (below dump) + also seen it with other indexes like index out of range [625] with length 625

2023/02/16 21:37:20 Starting bifrost-gateway 2023-02-16-8003960
2023/02/16 21:37:20 Block cache size: 1024
2023/02/16 21:37:20 Legacy RPC at /api/v0 provided by https://node0.delegate.ipfs.io https://node1.delegate.ipfs.io https://node2.delegate.ipfs.io https://node3.delegate.ipfs.io
2023/02/16 21:37:20 Path gateway listening on http://127.0.0.1:8081
2023/02/16 21:37:20 Subdomain gateway configured on dweb.link and http://localhost:8081
2023/02/16 21:37:20 Smoke test (JPG): http://127.0.0.1:8081/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
2023/02/16 21:37:20 Smoke test (Subdomain+DNSLink+UnixFS+HAMT): http://localhost:8081/ipns/en.wikipedia-on-ipfs.org/wiki/
2023/02/16 21:37:20 Metrics exposed at http://127.0.0.1:8041/debug/metrics/prometheus
2023-02-16T21:40:32.021Z    ERROR   core/server gateway/handler.go:305  A panic occurred in the gateway handler!
2023-02-16T21:40:32.021Z    ERROR   core/server gateway/handler.go:306  runtime error: index out of range [997] with length 997
goroutine 397 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
    /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/ipfs/go-libipfs/gateway.(*handler).ServeHTTP.func1()
    /go/pkg/mod/github.com/ipfs/go-libipfs@v0.5.1-0.20230215002322-e7aa6a184deb/gateway/handler.go:307 +0xea
panic({0xd12fa0, 0xc01fcb8510})
    /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/filecoin-saturn/caboose.(*pool).fetchWith(0xc0002ae900, {0x1150ed8, 0xc00acfe960}, {{0xc00acd6cc0?, 0x7fc68ced7128?}}, {0x0, 0x0})
    /go/pkg/mod/github.com/filecoin-saturn/caboose@v0.0.0-20230216184950-d2749d3b8402/pool.go:279 +0xbf2
github.com/filecoin-saturn/caboose.(*Caboose).Get(0xc000128ae0, {0x1150ed8, 0xc00acfe960}, {{0xc00acd6cc0?, 0xc00014c280?}})
    /go/pkg/mod/github.com/filecoin-saturn/caboose@v0.0.0-20230216184950-d2749d3b8402/caboose.go:116 +0xbb
main.(*exchangeBsWrapper).GetBlock(0xc000290490, {0x1150f10?, 0xc00d504180?}, {{0xc00acd6cc0?, 0xff?}})
    /go/src/github.com/ipfs/bifrost-gateway/blockstore.go:40 +0x1b5
github.com/ipfs/go-blockservice.getBlock({0x1150f10, 0xc00d504180}, {{0xc00acd6cc0?, 0x15?}}, {0x11566a8, 0xc00014c720}, 0xc00373adb8)
    /go/pkg/mod/github.com/ipfs/go-blockservice@v0.5.0/blockservice.go:248 +0x1a9
github.com/ipfs/go-blockservice.(*blockService).GetBlock(0xc0002fe9c0, {0x1150ed8, 0xc00acfe900}, {{0xc00acd6cc0?, 0xc00373af80?}})
    /go/pkg/mod/github.com/ipfs/go-blockservice@v0.5.0/blockservice.go:224 +0x319
main.(*bifrostGateway).GetBlock(0x414902?, {0x1150ed8?, 0xc00acfe900?}, {{0xc00acd6cc0?, 0x8500000000c38bc0?}})
    /go/src/github.com/ipfs/bifrost-gateway/gateway.go:114 +0x31
github.com/ipfs/go-libipfs/gateway.(*handler).handleGettingFirstBlock(0xc00028f500, 0xdadf3b?, {0xc00d2bb450?, 0xc00373b458?, 0x196a5e0?}, {0x1151140, 0xc01721d900}, {0x1155a90, 0xc021f72c00})
    /go/pkg/mod/github.com/ipfs/go-libipfs@v0.5.1-0.20230215002322-e7aa6a184deb/gateway/handler.go:897 +0xd6
github.com/ipfs/go-libipfs/gateway.(*handler).getOrHeadHandler(0xc00028f500, {0x7fc68c1cf760, 0xc00d2bb450}, 0xc00373b458)
    /go/pkg/mod/github.com/ipfs/go-libipfs@v0.5.1-0.20230215002322-e7aa6a184deb/gateway/handler.go:395 +0x7e5
github.com/ipfs/go-libipfs/gateway.(*handler).ServeHTTP(0xc00028f500, {0x7fc68c1cf760, 0xc00d2bb450}, 0xc0169f4d00)
    /go/pkg/mod/github.com/ipfs/go-libipfs@v0.5.1-0.20230215002322-e7aa6a184deb/gateway/handler.go:313 +0x211
net/http.(*ServeMux).ServeHTTP(0x1150e68?, {0x7fc68c1cf760, 0xc00d2bb450}, 0xc0169f4d00)
    /usr/local/go/src/net/http/server.go:2487 +0x149
github.com/ipfs/go-libipfs/gateway.WithHostname.func1({0x7fc68c1cf760, 0xc00d2bb450}, 0xc0169f4d00)
    /go/pkg/mod/github.com/ipfs/go-libipfs@v0.5.1-0.20230215002322-e7aa6a184deb/gateway/hostname.go:239 +0x2a5
net/http.HandlerFunc.ServeHTTP(0x1150250?, {0x7fc68c1cf760?, 0xc00d2bb450?}, 0xc00373b9b0?)
    /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1({0x1150250?, 0xc00ad5e7e0?}, 0xc0169f4d00)
    /go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:288 +0xc5
net/http.HandlerFunc.ServeHTTP(0xc00373ba50?, {0x1150250?, 0xc00ad5e7e0?}, 0xc00014c2a0?)
    /usr/local/go/src/net/http/server.go:2109 +0x2f
main.withRequestLogger.func1({0x1150250, 0xc00ad5e7e0}, 0xc0169f4d00)
    /go/src/github.com/ipfs/bifrost-gateway/handlers.go:38 +0x152
net/http.HandlerFunc.ServeHTTP(0xc00ad55cb9?, {0x1150250?, 0xc00ad5e7e0?}, 0x46936e?)
    /usr/local/go/src/net/http/server.go:2109 +0x2f
net/http.serverHandler.ServeHTTP({0xc0034ace40?}, {0x1150250, 0xc00ad5e7e0}, 0xc0169f4d00)
    /usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc0016c5b80, {0x1150f10, 0xc0009b5740})
    /usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:3102 +0x4db

filecoin-saturn / caboose

panic in pool.go:279 #29