Configuration
Mode: read
Workload: uniform
Timeout: 3m0s
Max error number at row: 1000
Max error number: unlimited
Retries:
number: 0
min interval: 80ms
max interval: 1s
handler: sb
Consistency level: quorum
Partition count: 60
Clustering rows: 10000000
Clustering row size: Fixed(2048)
Rows per request: 2000
Provide upper bound: false
IN queries: false
Order by: none
No lower bound: false
Page size: 1000
Concurrency: 700
Connections: 500
Maximum rate: 64000 op/s
Client compression: true
Hdr memory consumption: 344254400 bytes
2023/04/19 17:44:56 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[43 3491014] consistency=QUORUM] || ERROR: 0 attempts applied: Operation timed out for scylla_bench.test - received only 1 responses from 2 CL=QUORUM.
2023/04/19 17:44:56 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[39 3977063] consistency=QUORUM] || ERROR: 0 attempts applied: Operation timed out for scylla_bench.test - received only 1 responses from 2 CL=QUORUM.
time ops/s rows/s errors max 99.9th 99th 95th 90th median mean
2023/04/19 17:44:56 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[15 7113516] consistency=QUORUM] || ERROR: 0 attempts applied: Operation timed out for scylla_bench.test - received only 1 responses from 2 CL=QUORUM.
2023/04/19 17:44:56 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[55 3968802] consistency=QUORUM] || ERROR: 0 attempts applied: Operation timed out for scylla_bench.test - received only 1 responses from 2 CL=QUORUM.
...
<lots of similar timeout messages>
6s 1264 2528000 328 4.8s 4.7s 4.7s 4.4s 4.2s 370ms 1.1s
...
<lots of similar timeout messages>
...
2m0.1s 565 1131503 138 9.2s 8.7s 8.5s 8.3s 6s 18ms 1.3s
2023/04/19 17:44:57 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[58 5460126] consistency=QUORUM] || ERROR: 0 attempts applied: Operation timed out for scylla_bench.test - received only 1 responses from 2 CL=QUORUM.
...
<lots of similar timeout messages>
...
2023/04/19 17:52:43 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[35 5246275] consistency=QUORUM] || ERROR: 0 attempts applied: EOF
2023/04/19 17:52:44 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[35 1570023] consistency=QUORUM] || ERROR: 0 attempts applied: EOF
2023/04/19 17:52:44 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[40 1738314] consistency=QUORUM] || ERROR: 0 attempts applied: Operation failed for scylla_bench.test - received 0 responses and 1 failures from 2 CL=QUORUM.
2023/04/19 17:52:44 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[54 4939889] consistency=QUORUM] || ERROR: 0 attempts applied: gocql: connection closed waiting for response
2023/04/19 17:52:44 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[9 440632] consistency=QUORUM] || ERROR: 0 attempts applied: gocql: connection closed waiting for response
...
2023/04/19 17:54:44 [query statement="SELECT pk, ck, v FROM scylla_bench.test WHERE pk = ? AND ck >= ? LIMIT 2000 " values=[19 4463825] consistency=QUORUM] || ERROR: 0 attempts applied: Operation failed for scylla_bench.test - received 0 responses and 1 failures from 2 CL=QUORUM.
2023/04/19 17:56:26 error: failed to connect to "[HostInfo hostname=\"10.4.2.156\" connectAddress=\"10.4.2.156\" peer=\"10.4.2.156\" rpc_address=\"10.4.2.156\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.4.2.156\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"71ffdcba-9a8e-4cfd-8d68-ec36e4250afa\" version=\"v3.0.8\" state=DOWN num_tokens=256]" due to error: gocql: no response to connection startup within timeout
2023/04/19 17:56:59 error: failed to connect to "[HostInfo hostname=\"10.4.2.156\" connectAddress=\"10.4.2.156\" peer=\"10.4.2.156\" rpc_address=\"10.4.2.156\" broadcast_address=\"<nil>\" preferred_ip=\"<nil>\" connect_addr=\"10.4.2.156\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"eu-west\" rack=\"1a\" host_id=\"71ffdcba-9a8e-4cfd-8d68-ec36e4250afa\" version=\"v3.0.8\" state=DOWN num_tokens=256]" due to error: gocql: no response to connection startup within timeout
Killed
Note that the lost connection to the 10.4.2.156 node is expected, because it is result of the soft_reboot_node nemesis:
scylla-bench:
scylladb/hydra-loaders:scylla-bench-v0.1.16(docker) scylla:2023.1.0~rc4-0.20230402.f90af81a13cc with build-id 4db7799d283c7ca63c7639b04db34c301dbcfb66(non-docker) Platform:AWS/eu-west-1DB nodes num:4DB node instance type:i3en.3xlargeLoaders num:3Loader node instance type:c5n.2xlargeLoader commands:
So, running above commands one on a loader we got one of them (
loader-3) OOMkilled:And here is how
loader-3logs looked like:Note that the lost connection to the
10.4.2.156node is expected, because it is result of thesoft_reboot_nodenemesis:So, probably we have some kind of memory leak here.
Logs: