panic: runtime error: invalid memory address or nil pointer dereference

[steve-marmalade]

The imagor service has been crashing periodically with the following error message:

panic: runtime error: invalid memory address or nil pointer dereference
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x64
golang.org/x/sync/singleflight.newPanicError({0xe45820?, 0x1a42860})
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:35 +0x28
golang.org/x/sync/singleflight.(*Group).doCall.func2.1()
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:184 +0x40
panic({0xe45820, 0x1a42860})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
cloud.google.com/go/storage.(*Reader).Read(0x0, {0x400289ec00, 0x4002505288?, 0x200})
        /go/pkg/mod/cloud.google.com/go/storage@v1.31.0/reader.go:215 +0x24
io.ReadAtLeast({0xfb7a562c4520, 0x0}, {0x400289ec00, 0x200, 0x200}, 0x200)
        /usr/local/go/src/io/io.go:332 +0xa0
github.com/cshum/imagor.(*Blob).doInit(0x4000a90090)
        /go/src/github.com/cshum/imagor/blob.go:286 +0x388
sync.(*Once).doSlow(0x0?, 0x0?)
        /usr/local/go/src/sync/once.go:74 +0x100
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
github.com/cshum/imagor.(*Blob).init(...)
        /go/src/github.com/cshum/imagor/blob.go:232
github.com/cshum/imagor.(*Blob).Err(0x4000a90090?)
        /go/src/github.com/cshum/imagor/blob.go:507 +0x54
github.com/cshum/imagor.checkBlob(...)
        /go/src/github.com/cshum/imagor/blob.go:517
github.com/cshum/imagor.(*Imagor).fromStoragesAndLoaders(0x0?, 0x4002871a00, {0x40003eb5b0, 0x1, 0x1}, {0x40003eb610, 0x1, 0x0?}, {0x400286e050, 0x43})
        /go/src/github.com/cshum/imagor/imagor.go:514 +0x254
github.com/cshum/imagor.(*Imagor).loadStorage(0x4000323d40, 0x44655c?, {0x400286e050, 0x43})
        /go/src/github.com/cshum/imagor/imagor.go:483 +0x70
github.com/cshum/imagor.(*Imagor).Do.func2({0x1312268, 0x4001d9f4d0}, 0x4002c7fb10)
        /go/src/github.com/cshum/imagor/imagor.go:359 +0x5b8
github.com/cshum/imagor.(*Imagor).suppress.func2()
        /go/src/github.com/cshum/imagor/imagor.go:613 +0xa8
golang.org/x/sync/singleflight.(*Group).doCall.func2(0x4002505f46, 0x4003b4f8b0, 0x400286a000?)
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:189 +0x60
golang.org/x/sync/singleflight.(*Group).doCall(0x0?, 0x4002948f58?, {0x40026aac30?, 0x46f0f4?}, 0x400233b9e0?)
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:191 +0x84
created by golang.org/x/sync/singleflight.(*Group).DoChan
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:129 +0x388
goroutine 332790 [running]:
golang.org/x/sync/singleflight.(*Group).doCall.func1.2()
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:158 +0x2c
created by golang.org/x/sync/singleflight.(*Group).doCall.func1
        /go/pkg/mod/golang.org/x/sync@v0.3.0/singleflight/singleflight.go:158 +0x2ac

A couple of relevant notes:

• The crash came at a surge in traffic, which peaked at 41 req/s to this single-core instance (significantly in excess of the 2 * n_cpu advice I've seen given in other issues). The reason for the concurrency is that I am currently scaling based on target CPU utilization rather than number of connections (and CPU utilization maxed out at < 40% during this surge). Let me know if you think this is unwise / could be the cause. If it is, I wonder if you have any advice on how I can run imagor in such a way that allows for higher CPU utilization?
• As an experiment, I am running on Ampere Altra-based ARM VMs (which required me to build the imagor Docker container for the linux/arm64 platform). While things generally appear to be working as expected, I'm noting it here in case it raises any red flags.