grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.65k stars 576 forks source link

Panic during Ad Hoc profile upload #3320

Closed aleks-p closed 1 month ago

aleks-p commented 1 month ago

I've observed the following panic today:

connectrpc.com/connect.NewUnaryHandler[...].func1({0x2f22c70, 0xc0001e2100})
    github.com/grafana/pyroscope/pkg/adhocprofiles/adhocprofiles.go:78 +0x2a5
github.com/grafana/pyroscope/pkg/adhocprofiles.(*AdHocProfiles).Upload(0xc000afc480, {0x2f14e78, 0xc000036000}, 0xc0001e2100)
    github.com/grafana/pyroscope/pkg/adhocprofiles/adhocprofiles.go:224 +0x105
github.com/grafana/pyroscope/pkg/adhocprofiles.parse(0xc000d3ef28, 0x0, 0xc000d3eee0?)
    github.com/grafana/pyroscope/pkg/og/structs/flamebearer/convert/convert.go:63 +0x8a
github.com/grafana/pyroscope/pkg/og/structs/flamebearer/convert.FlamebearerFromFile({{0xc000156708, 0x11}, {0x0, 0x0}, {{0x0, 0x0}, {0x0, 0x0}}, {0xc00138c000, 0x38d, ...}}, ...)
    github.com/grafana/pyroscope/pkg/og/structs/flamebearer/convert/convert.go:74 +0x39
github.com/grafana/pyroscope/pkg/og/structs/flamebearer/convert.Converter.func1({0xc00138c000?, 0x11?, 0x0?}, {0xc000156708?, 0x0?}, 0x0?)
    github.com/grafana/pyroscope/pkg/og/structs/flamebearer/convert/convert.go:169 +0x178
github.com/grafana/pyroscope/pkg/og/structs/flamebearer/convert.JSONToProfile({0xc00138c000, 0x38d, 0x38d}, {0x0?, 0x0?}, 0x2000)
    github.com/grafana/pyroscope/pkg/og/structs/flamebearer/flamebearer.go:347 +0x11e
github.com/grafana/pyroscope/pkg/og/structs/flamebearer.ProfileToTree({0x1, {{{0xc0001e2180, 0x6, 0x8}, {0xc000f9c000, 0x3, 0x4}, 0x64, 0x3c}, {{0xc000a98100, ...}, ...}, ...}, ...})
    github.com/grafana/pyroscope/pkg/og/structs/flamebearer/flamebearer.go:361 +0x1c5
github.com/grafana/pyroscope/pkg/og/structs/flamebearer.flamebearerV1ToTree({{0xc0001e2180, 0x6, 0x8}, {0xc000f9c000, 0x3, 0x4}, 0x64, 0x3c})
    github.com/grafana/pyroscope/pkg/og/structs/flamebearer/flamebearer.go:387 +0x350
github.com/grafana/pyroscope/pkg/og/structs/flamebearer.buildStack({{0xc0001e2180, 0x6, 0x8}, {0xc000f9c000, 0x3, 0x4}, 0x64, 0x3c}, 0x2, 0x0)
    runtime/panic.go:914 +0x21f
panic({0x2264420?, 0xc0008722e8?})
    github.com/grafana/pyroscope/pkg/util/recovery.go:69 +0x2f
github.com/grafana/pyroscope/pkg/util.(*recoveryInterceptor).WrapUnary.recoveryInterceptor.WrapUnary.func1.1()
    github.com/grafana/pyroscope/pkg/util/recovery.go:44 +0x52
github.com/grafana/pyroscope/pkg/util.panicError({0x2264420?, 0xc0008722e8})
goroutine 970866 [running]:
panic: runtime error: index out of range [-1]

Seems to happen in https://github.com/grafana/pyroscope/blob/1a3a8a176156ed7ca561cb7398b146259f9eb0de/pkg/og/structs/flamebearer/flamebearer.go#L387