search

scylladb / scylla-bench

43 stars 36 forks source link

panic: runtime error: index out of range [2] with length 0 #122

Open vponomaryov opened 1 year ago

vponomaryov commented 1 year ago

Issue description

Describe your issue in detail and steps it took to produce it.

Impact

It is unclear what a user does wrong or can do with it.

How frequently does it reproduce?

First time observed.

Installation details

Kernel Version: 5.15.0-1028-aws Scylla version (or git commit hash): 2022.2.0-20230112.4f0f82ff2e1d with build-id 6bb071079708f1e48d1faeee6ef3cc3ef93733af

Cluster size: 4 nodes (is4gen.4xlarge)

Scylla Nodes used in this run:

OS / Image: ami-0a018bb5789118823 (aws: eu-west-1)

Test: longevity-large-partition-4days-arm-test Test id: b4fd1150-5888-4a5c-b35a-6f0e3fdbdad9 Test name: scylla-5.2/longevity/longevity-large-partition-4days-arm-test Test config file(s):

Details:

Running large partitions test where we have huge latencies appeared following error:

panic: runtime error: index out of range [2] with length 0

goroutine 1 [running]:
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).getCountAtIndex(...)
    /go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:595
github.com/HdrHistogram/hdrhistogram-go.(*iterator).nextCountAtIdx(0x10000203073?, 0x7ff01ad9d748?)
    /go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:662 +0xc5
github.com/HdrHistogram/hdrhistogram-go.(*iterator).next(0xc0000ef020)
    /go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:670 +0x25
github.com/HdrHistogram/hdrhistogram-go.(*rIterator).next(...)
    /go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:683
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).Merge(0xf0000000e?, 0x4000000000a?)
    /go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:177 +0x8d
github.com/scylladb/scylla-bench/pkg/results.(*MergedResult).AddResult(0xc1cfb01bc0, {0x0, 0x0, 0x1, 0x7d0, 0x0, {0x0, 0x0, 0x0}, 0xc2f5d83300, ...})
    /go/scylla-bench-0.1.16/pkg/results/merged_result.go:54 +0x1b5
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetResultsFromThreadsAndMerge(0xc000162840)
    /go/scylla-bench-0.1.16/pkg/results/thread_results.go:60 +0x89
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetTotalResults(0xc000162840)
    /go/scylla-bench-0.1.16/pkg/results/thread_results.go:82 +0xcc
main.main()
    /go/scylla-bench-0.1.16/main.go:640 +0x3b1e

Probably appears having a too big loader load like in the following bug: https://github.com/scylladb/scylla-bench/issues/121

Monitoring:

Logs:

Jenkins job URL

fruch commented 1 year ago

Probably appears having a too big loader load like in the following bug: https://github.com/scylladb/scylla-bench/issues/121

you mean we need a bigger loader ? I would rather collect some of the stats/graphs to show it's actually the case, before picking up bigger loader nodes

vponomaryov commented 1 year ago

Probably appears having a too big loader load like in the following bug: #121

you mean we need a bigger loader ? I would rather collect some of the stats/graphs to show it's actually the case, before picking up bigger loader nodes

In good case the s-b bugs must be fixed. I just described the observation that the bigger instance types reduce probability of such s-b crashes.