Loader got OOM running a scylla-bench read stress

Issue description

[ ] This issue is a regression.
[ ] It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

loader-4 ran the following s-b read stress for ~ 12.5 hours (out of expected 24 hours).

Its log file is: scylla-bench-l0-bdfd47ae-45c4-4348-a835-ba48fe0397b9.log

SCT log shows:

2023_07_29__03_05_15_387.sct-5cd89f45.log:&lt; t:2023-07-29 07:09:25,838 f:scylla_bench_thread.py l:195  c:sdcm.scylla_bench_thread p:DEBUG &gt; Scylla bench command: scylla-bench -workload=sequential -mode=read -replication-factor=3 -partition-count=1000 -clustering-row-count=200000 -clustering-row-size=uniform:10..10240 -rows-per-request=10 -consistency-level=quorum -timeout=90s -partition-offset=1001 -concurrency=100 -connection-count=100 -iterations=0 -duration=1440m  -error-at-row-limit 1000
2023_07_29__03_05_15_387.sct-5cd89f45.log:&lt; t:2023-07-29 07:09:25,840 f:remote_base.py  l:520  c:RemoteLibSSH2CmdRunner p:DEBUG &gt; Running command "sudo  docker exec e9a633737a5dc44efd1e27efe37cceba012565f98f333d7a4828a9ae42f4b1e7 /bin/sh -c 'scylla-bench -workload=sequential -mode=read -replication-factor=3 -partition-count=1000 -clustering-row-count=200000 -clustering-row-size=uniform:10..10240 -rows-per-request=10 -consistency-level=quorum -timeout=90s -partition-offset=1001 -concurrency=100 -connection-count=100 -iterations=0 -duration=1440m  -error-at-row-limit 1000 -nodes 10.142.0.134,10.142.0.143,10.142.0.147,10.142.0.152,10.142.0.8'"...
2023_07_29__03_05_15_387.sct-5cd89f45.log:&lt; t:2023-07-29 07:09:25,841 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  &gt; stress_cmd=scylla-bench -workload=sequential -mode=read -replication-factor=3 -partition-count=1000 -clustering-row-count=200000 -clustering-row-size=uniform:10..10240 -rows-per-request=10 -consistency-level=quorum -timeout=90s -partition-offset=1001 -concurrency=100 -connection-count=100 -iterations=0 -duration=1440m  -error-at-row-limit 1000 -nodes 10.142.0.134,10.142.0.143,10.142.0.147,10.142.0.152,10.142.0.8
2023_07_29__19_27_20_757.sct-5cd89f45.log:&lt; t:2023-07-29 19:39:25,215 f:base.py         l:146  c:RemoteLibSSH2CmdRunner p:ERROR &gt; Error executing command: "sudo  docker exec e9a633737a5dc44efd1e27efe37cceba012565f98f333d7a4828a9ae42f4b1e7 /bin/sh -c 'scylla-bench -workload=sequential -mode=read -replication-factor=3 -partition-count=1000 -clustering-row-count=200000 -clustering-row-size=uniform:10..10240 -rows-per-request=10 -consistency-level=quorum -timeout=90s -partition-offset=1001 -concurrency=100 -connection-count=100 -iterations=0 -duration=1440m  -error-at-row-limit 1000 -nodes 10.142.0.134,10.142.0.143,10.142.0.147,10.142.0.152,10.142.0.8'"; Exit status: 137

The grafana shows: Screenshot from 2023-08-02 19-51-34

Installation details

Kernel Version: 5.15.0-1038-gcp Scylla version (or git commit hash): 5.4.0~dev-20230728.7351c8424df3 with build-id d47e513b9bff8db5782a5220c05f5ac2a70b7a6c

Cluster size: 5 nodes (n2-highmem-16)

Scylla Nodes used in this run:

longevity-large-partitions-200k-pks-db-node-5cd89f45-0-9 (35.227.121.146 | 10.142.0.38) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-8 (34.74.86.71 | 10.142.0.10) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-7 (34.75.32.50 | 10.142.0.12) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-6 (35.227.121.225 | 10.142.0.8) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-5 (34.75.69.205 | 10.142.0.155) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-4 (34.73.170.235 | 10.142.0.152) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-3 (34.75.238.169 | 10.142.0.147) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-2 (34.23.248.117 | 10.142.0.143) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-13 (35.231.234.211 | 10.142.0.64) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-12 (34.148.129.203 | 10.142.0.63) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-11 (35.231.234.211 | 10.142.0.42) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-10 (34.23.248.117 | 10.142.0.39) (shards: 14)
longevity-large-partitions-200k-pks-db-node-5cd89f45-0-1 (35.231.157.23 | 10.142.0.134) (shards: 14)

OS / Image: `` (gce: undefined_region)

Test: longevity-large-partition-200k-pks-4days-gce-test Test id: 5cd89f45-07f3-4865-8f4a-34c4071e2bdd Test name: scylla-master/longevity/longevity-large-partition-200k-pks-4days-gce-test Test config file(s):

longevity-large-partition-200k_pks-4days.yaml

Logs and commands

- Restore Monitor Stack command: `$ hydra investigate show-monitor 5cd89f45-07f3-4865-8f4a-34c4071e2bdd` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=5cd89f45-07f3-4865-8f4a-34c4071e2bdd) - Show all stored logs command: `$ hydra investigate show-logs 5cd89f45-07f3-4865-8f4a-34c4071e2bdd` ## Logs: - **db-cluster-5cd89f45.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/db-cluster-5cd89f45.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/db-cluster-5cd89f45.tar.gz) - **debug-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/debug-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/debug-5cd89f45.log.tar.gz) - **raw_events-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/raw_events-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/raw_events-5cd89f45.log.tar.gz) - **summary-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/summary-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/summary-5cd89f45.log.tar.gz) - **error-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/error-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/error-5cd89f45.log.tar.gz) - **critical-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/critical-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/critical-5cd89f45.log.tar.gz) - **output-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/output-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/output-5cd89f45.log.tar.gz) - **warning-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/warning-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/warning-5cd89f45.log.tar.gz) - **argus-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/argus-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/argus-5cd89f45.log.tar.gz) - **normal-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/normal-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/normal-5cd89f45.log.tar.gz) - **events-5cd89f45.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/events-5cd89f45.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/events-5cd89f45.log.tar.gz) - **2023_07_29__03_05_15_387.sct-5cd89f45.log.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_29__03_05_15_387.sct-5cd89f45.log.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_29__03_05_15_387.sct-5cd89f45.log.gz) - **2023_07_29__13_08_54_529.sct-5cd89f45.log.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_29__13_08_54_529.sct-5cd89f45.log.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_29__13_08_54_529.sct-5cd89f45.log.gz) - **2023_07_29__19_27_20_757.sct-5cd89f45.log.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_29__19_27_20_757.sct-5cd89f45.log.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_29__19_27_20_757.sct-5cd89f45.log.gz) - **2023_07_30__02_39_44_655.sct-5cd89f45.log.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_30__02_39_44_655.sct-5cd89f45.log.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/2023_07_30__02_39_44_655.sct-5cd89f45.log.gz) - **loader-set-5cd89f45.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/loader-set-5cd89f45.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/loader-set-5cd89f45.tar.gz) - **monitor-set-5cd89f45.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/monitor-set-5cd89f45.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/5cd89f45-07f3-4865-8f4a-34c4071e2bdd/20230730_091441/monitor-set-5cd89f45.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/longevity/job/longevity-large-partition-200k-pks-4days-gce-test/7/) [Argus](https://argus.scylladb.com/test/917e825f-11f9-4493-acdb-ec5266a3af78/runs?additionalRuns[]=5cd89f45-07f3-4865-8f4a-34c4071e2bdd)

scylladb / scylla-bench