search

grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.98k stars 598 forks source link

symdb: Panic during ResolveStacktraceLocations #3406

Closed simonswine closed 2 months ago

simonswine commented 3 months ago

Internally at GL, we observed those panics on at least two ingesters that gotten querried:

{"from":"2024-07-05 08:22:16","to":"2024-07-05 08:53:12"}
  |   | 2024-07-05 08:47:07.580 |   | created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 223338 |  
  |   | 2024-07-05 08:47:07.580 |   | golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x56 |  
  |   | 2024-07-05 08:47:07.580 |   | golang.org/x/sync/errgroup.(*Group).Go.func1() |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/util/recovery.go:59 +0x58 |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).withSymbols.RecoverPanic.func2() |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:279 +0x5e |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).withSymbols.func1() |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:238 +0x45 |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Resolver).Tree.func1(0xc00409ba40?, 0x2f5f598?) |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver.go:299 |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*Symbols).Tree(...) |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver_tree.go:40 +0x442 |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.buildTree({0x2f5f598, 0xc001ccb110}, 0xc01db70850, 0xc0211e9680, 0x4000) |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/block_reader.go:398 +0x12d |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*partition).ResolveStacktraceLocations(0xc00409ba40, {0x2f5f598?, 0xc001ccb110}, {0x2f47ec0?, 0xc031a0a240}, {0xc0043777e8?, 0x78dcdedbb8d8?, 0x78dd26451498?}) |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/block_reader.go:465 +0xf5 |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*stacktracesLookup).do(0xc08adf5ce8) |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb/resolver_tree.go:89 +0x1db |  
  |   | 2024-07-05 08:47:07.580 |   | github.com/grafana/pyroscope/pkg/phlaredb/symdb.(*treeSymbols).InsertStacktrace(0xc031a0a240, 0x1?, {0xc006c62200, 0x3, 0x1?})
panic: runtime error: index out of range [35473] with length 0