search

scylladb / scylla-bench

42 stars 34 forks source link

scylla-bench v0.1.8 got fatal error: runtime: out of memory #89

Open yarongilor opened 2 years ago

yarongilor commented 2 years ago

Installation details Kernel version: 5.11.0-1022-aws Scylla version (or git commit hash): 4.6.rc5-0.20220203.5694ec189 with build-id f5d85bf5abe6d2f9fd3487e2469ce1c34304cc14 Cluster size: 3 3 3 nodes (i3.2xlarge) Scylla running with shards number (live nodes): longevity-counters-multidc-4-6-db-node-ec7ffc0b-1 (34.254.228.100 | 10.0.0.60): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-2 (52.19.209.202 | 10.0.3.34): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-3 (54.75.1.178 | 10.0.2.127): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-4 (52.34.214.217 | 10.0.3.8): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-5 (35.88.112.46 | 10.0.2.163): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-6 (35.88.221.142 | 10.0.2.18): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-7 (44.200.34.43 | 10.0.3.238): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-8 (44.195.21.232 | 10.0.0.201): 8 shards longevity-counters-multidc-4-6-db-node-ec7ffc0b-9 (3.228.17.3 | 10.0.3.20): 8 shards OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-06df116ea1c70171c ami-0d2bccabf736b9dd0 ami-097f7b2efb7723718 (aws: eu-west-1 us-west-2 us-east-1)

Test: longevity-counters-multidc-test Test name: longevity_test.LongevityTest.test_custom_time Test config file(s):

Issue description

====================================

scylla-bench got the following OOM with version 0.1.8: from the loader-set log of scylla-bench-l0-48d0dafb-2e83-46b1-8add-6e7dc9fc431c.log: (./longevity-counters-multidc-4-6-loader-node-ec7ffc0b-1/scylla-bench-l0-48d0dafb-2e83-46b1-8add-6e7dc9fc431c.log)

2022/02/06 16:51:00 writev tcp 10.0.2.255:39009->35.88.221.142:9042: i/o timeout
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x6fbe3b, 0x400000})
    /usr/local/go/src/runtime/panic.go:1198 +0x71
runtime.sysMap(0xc5cf800000, 0x1, 0xc000080800)
    /usr/local/go/src/runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x93c080, 0x43bdef)
    /usr/local/go/src/runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x93c080, 0x1, 0x0, 0x12)
    /usr/local/go/src/runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
    /usr/local/go/src/runtime/mheap.go:913 +0x69
runtime.systemstack()
    /usr/local/go/src/runtime/asm_amd64.s:383 +0x49

goroutine 671 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc00c367820 sp=0xc00c367818 pc=0x462220
runtime.(*mheap).alloc(0x761f50, 0xc0000180b0, 0x68, 0x1)
    /usr/local/go/src/runtime/mheap.go:907 +0x73 fp=0xc00c367870 sp=0xc00c367820 pc=0x426ed3
runtime.(*mcentral).grow(0x2000)
    /usr/local/go/src/runtime/mcentral.go:241 +0x65 fp=0xc00c3678b8 sp=0xc00c367870 pc=0x418865
runtime.(*mcentral).cacheSpan(0x94d048)
    /usr/local/go/src/runtime/mcentral.go:161 +0x69e fp=0xc00c367930 sp=0xc00c3678b8 pc=0x41869e
runtime.(*mcache).refill(0x7febc181d5b8, 0x12)
    /usr/local/go/src/runtime/mcache.go:162 +0xaf fp=0xc00c367980 sp=0xc00c367930 pc=0x4178cf
runtime.(*mcache).nextFree(0x7febc181d5b8, 0x12)
    /usr/local/go/src/runtime/malloc.go:880 +0x85 fp=0xc00c3679c8 sp=0xc00c367980 pc=0x40da85
runtime.mallocgc(0x70, 0x6a3580, 0x1)
    /usr/local/go/src/runtime/malloc.go:1071 +0x4e8 fp=0xc00c367a48 sp=0xc00c3679c8 pc=0x40e108
runtime.newobject(0xc5cf771b90)
    /usr/local/go/src/runtime/malloc.go:1228 +0x27 fp=0xc00c367a70 sp=0xc00c367a48 pc=0x40e627
main.DoReadsFromTable.func1(0xc0003bc990)
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:479 +0x78e fp=0xc00c367c78 sp=0xc00c367a70 pc=0x680bee
main.RunTest(0xc0003bc990, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80}, 0xc00c367e40)
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:126 +0x1b0 fp=0xc00c367db0 sp=0xc00c367c78 pc=0x67d5d0
main.DoReadsFromTable({0x6f8ac5, 0xd}, 0x0, 0x0, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80})
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:427 +0x16e fp=0xc00c367ea0 sp=0xc00c367db0 pc=0x6802ce
main.DoCounterReads(0x7ffe23e16d11, 0x0, {0x763ad0, 0xc00020c570}, {0x75f8a0, 0x952d80})
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:384 +0x53 fp=0xc00c367ef0 sp=0xc00c367ea0 pc=0x67f9f3
main.main.func4(0x0, 0x0, {0x75f8a0, 0x952d80})
    /tmp/sb_install/scylla-bench-0.1.8/main.go:527 +0x105 fp=0xc00c367f88 sp=0xc00c367ef0 pc=0x67c7e5
main.RunConcurrently.func1(0x5c)
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:78 +0x66 fp=0xc00c367fc8 sp=0xc00c367f88 pc=0x67d326
main.RunConcurrently·dwrap·2()
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:79 +0x2d fp=0xc00c367fe0 sp=0xc00c367fc8 pc=0x67d28d
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00c367fe8 sp=0xc00c367fe0 pc=0x464441
created by main.RunConcurrently
    /tmp/sb_install/scylla-bench-0.1.8/modes.go:76 +0x10c

goroutine 1 [runnable]:
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).getCountAtIndex(...)
    /home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:595
github.com/HdrHistogram/hdrhistogram-go.(*iterator).nextCountAtIdx(0x4e714e7c6b18b182, 0x630d02662c804285)
    /home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:662 +0x65
github.com/HdrHistogram/hdrhistogram-go.(*iterator).next(0xc000107530)
    /home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:670 +0x25
github.com/HdrHistogram/hdrhistogram-go.(*rIterator).next(...)
    /home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:683
github.com/HdrHistogram/hdrhistogram-go.(*Histogram).Merge(0xf00000009, 0x4000000000a)
    /home/centos/go/pkg/mod/github.com/!hdr!histogram/hdrhistogram-go@v1.1.2/hdr.go:177 +0x98
github.com/scylladb/scylla-bench/pkg/results.(*MergedResult).AddResult(0xc07adacba0, {0x0, 0x0, 0x6, 0x6, 0x0, {0x0, 0x0, 0x0}, 0xc09f0c2600, ...})
    /tmp/sb_install/scylla-bench-0.1.8/pkg/results/merged_result.go:53 +0x1b0
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetResultsFromThreadsAndMerge(0xc0002e6600)
    /tmp/sb_install/scylla-bench-0.1.8/pkg/results/thread_results.go:60 +0x89
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetTotalResults(0xc0002e6600)
    /tmp/sb_install/scylla-bench-0.1.8/pkg/results/thread_results.go:82 +0xcd
main.main()
    /tmp/sb_install/scylla-bench-0.1.8/main.go:530 +0x31f7

goroutine 80 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020ebc0})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7feb9848b038, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c400, 0xc000144000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c400, {0xc000144000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c400, {0xc000144000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010038, {0xc000144000, 0x123, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c300, {0xc000146048, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c300}, {0xc000146048, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c300}, {0xc000146048, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146000, {0x761f18, 0xc00005a240})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a240})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 79 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a558, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009cc00, 0xc000250000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009cc00, {0xc000250000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009cc00, {0xc000250000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0c8, {0xc000250000, 0x40, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002018c0, {0xc00023a4c8, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc0002018c0}, {0xc00023a4c8, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc0002018c0}, {0xc00023a4c8, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a480, {0x761f18, 0xc00020ebc0})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0xc0000f00e0, {0x761f18, 0xc00020ebc0})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 51 [select, 2 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc00019a190)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:39 +0x67
created by github.com/gocql/gocql.newEventDebouncer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:27 +0x11f

goroutine 52 [select, 54 minutes]:
github.com/gocql/gocql.(*eventDebouncer).flusher(0xc00019a230)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:39 +0x67
created by github.com/gocql/gocql.newEventDebouncer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/events.go:27 +0x11f

goroutine 72 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a380})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 37 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00009ab40})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 26 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a240})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 35 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002008a0, 0xc000187480)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:874 +0x1b9
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 36 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848af50, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c580, 0xc0000f6000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c580, {0xc0000f6000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c580, {0xc0000f6000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0000960f8, {0xc0000f6000, 0x41, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00008eba0, {0xc0000f8048, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00008eba0}, {0xc0000f8048, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00008eba0}, {0xc0000f8048, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc0000f8000, {0x761f18, 0xc00009ab40})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00009ab40})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 39 [select]:
github.com/gocql/gocql.(*controlConn).heartBeat(0x0)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/control.go:70 +0x10d
created by github.com/gocql/gocql.(*controlConn).connect
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/control.go:259 +0x13b

goroutine 75 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a280})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 529 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005be80})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 86 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a400})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 69 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e6c0})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 73 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc000201800, 0xc00005a380)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:874 +0x1b9
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 48 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002014a0, 0x0)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 98 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00005d140, 0x0)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 84 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00008f560, 0x0)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 67 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002015c0, 0xc00020e700)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 70 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc0002016e0, 0xc00020e6c0)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 33 [select]:
github.com/gocql/gocql.(*writeCoalescer).writeFlusher(0xc00008f440, 0xc00002df88)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:866 +0x13e
created by github.com/gocql/gocql.newWriteCoalescer
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:771 +0x17a

goroutine 74 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ae68, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c680, 0xc00014a000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c680, {0xc00014a000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c680, {0xc00014a000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010048, {0xc00014a000, 0x3f, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c480, {0xc0001461c8, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c480}, {0xc0001461c8, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c480}, {0xc0001461c8, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146180, {0x761f18, 0xc00005a280})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a280})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 99 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ac98, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000212380, 0xc000238000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000212380, {0xc000238000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000212380, {0xc000238000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0a8, {0xc000238000, 0x42, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000200cc0, {0xc00023a048, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc000200cc0}, {0xc00023a048, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc000200cc0}, {0xc00023a048, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a000, {0x761f18, 0xc00020e640})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e640})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 49 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848ad80, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00018c700, 0xc00014e000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00018c700, {0xc00014e000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00018c700, {0xc00014e000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010050, {0xc00014e000, 0x3e, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c720, {0xc000146348, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c720}, {0xc000146348, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c720}, {0xc000146348, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146300, {0x761f18, 0xc00005a300})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a300})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 68 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848aac8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00018c800, 0xc00023e000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00018c800, {0xc00023e000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00018c800, {0xc00023e000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0b0, {0xc00023e000, 0x3f, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc000200f60, {0xc00023a1c8, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc000200f60}, {0xc00023a1c8, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc000200f60}, {0xc00023a1c8, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a180, {0x761f18, 0xc00020e6c0})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e6c0})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 82 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a8f8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c900, 0xc000242000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c900, {0xc000242000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c900, {0xc000242000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020a0b8, {0xc000242000, 0x3e, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0002010e0, {0xc00023a348, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc0002010e0}, {0xc00023a348, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc0002010e0}, {0xc00023a348, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc00023a300, {0x761f18, 0xc00020e700})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00020e700})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 71 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848a9e0, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000212480, 0xc000158000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000212480, {0xc000158000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000212480, {0xc000158000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010058, {0xc000158000, 0x41, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005c9c0, {0xc0001464c8, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005c9c0}, {0xc0001464c8, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005c9c0}, {0xc0001464c8, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146480, {0x761f18, 0xc00005a380})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a380})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 85 [runnable]:
internal/poll.runtime_pollWait(0x7feb9848abb0, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00009c780, 0xc00015a000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00009c780, {0xc00015a000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc00009c780, {0xc00015a000, 0x407029, 0x6b42a0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000010060, {0xc00015a000, 0x40, 0x1})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00005cb40, {0xc000146648, 0x1, 0x0})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x75b9e0, 0xc00005cb40}, {0xc000146648, 0x1, 0x9}, 0x1)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
github.com/gocql/gocql.readHeader({0x75b9e0, 0xc00005cb40}, {0xc000146648, 0x9, 0x9})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/frame.go:452 +0xa5
github.com/gocql/gocql.(*Conn).recv(0xc000146600, {0x761f18, 0xc00005a400})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:645 +0xed
github.com/gocql/gocql.(*Conn).serve(0x0, {0x761f18, 0xc00005a400})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:562 +0x2e
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:312 +0x248

goroutine 66 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00005a300})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 83 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e700})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 100 [select]:
github.com/gocql/gocql.(*Conn).heartBeat(0x0, {0x761f18, 0xc00020e640})
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:602 +0x125
created by github.com/gocql/gocql.(*Conn).init
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/conn.go:313 +0x2af

goroutine 332 [select]:
github.com/gocql/gocql.(*Session).reconnectDownedHosts(0xc0001ba380, 0x0)
    /home/centos/go/pkg/mod/github.com/scylladb/gocql@v1.5.0/session.go:321 +0x106
created by github.com/gocql/gocql.(*Session).init

====================================

Restore Monitor Stack command: $ hydra investigate show-monitor ec7ffc0b-b344-43db-9777-6db183568e53 Restore monitor on AWS instance using Jenkins job Show all stored logs command: $ hydra investigate show-logs ec7ffc0b-b344-43db-9777-6db183568e53

Test id: ec7ffc0b-b344-43db-9777-6db183568e53

Logs: grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png](https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-longevity-counters-multidc-test-scylla-per-server-metrics-nemesis-20220206_172345-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png) grafana - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png](https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_172133/grafana-screenshot-overview-20220206_172133-longevity-counters-multidc-4-6-monitor-node-ec7ffc0b-1.png) db-cluster - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/db-cluster-ec7ffc0b.tar.gz) loader-set - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/loader-set-ec7ffc0b.tar.gz) monitor-set - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/monitor-set-ec7ffc0b.tar.gz) sct - [https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/ec7ffc0b-b344-43db-9777-6db183568e53/20220206_173419/sct-runner-ec7ffc0b.tar.gz)

Jenkins job URL