Add testcase for data being deleted using compaction while having 90% storage utilization

[pehala]

• Create 3 node cluster with rf=3.
• Reaching 90% disk usage.
• Keep it running for an hour, in-between we should clean up some data by dropping tables and keyspace.
• Wait for compaction

[cezarmoise]

Results

Initial Cluster: 4 x i4i.large with auto_snapshot: false Write to 70%; Sleep 30 min Write to 90%; Sleep 30 min Writes have RF=3 Drop some tables Sleep 30 minutes

---

reached 91% disk usage and started waiting for 30mins, no write or read.

16:59:51  < t:2024-11-14 14:59:50,996 f:full_storage_utilization_test.py l:166  c:FullStorageUtilizationTest2 p:INFO  > Current max disk usage after writing to keyspace14: 91% (395 GB / 392.40000000000003 GB)
16:59:53  < t:2024-11-14 14:59:53,007 f:full_storage_utilization_test.py l:134  c:FullStorageUtilizationTest2 p:INFO  > Wait for 1800 seconds

After 30min idle time, started dropping some tables.

17:31:02  < t:2024-11-14 15:31:02,190 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_large3.standard1
17:41:05  < t:2024-11-14 15:41:03,922 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_large4.standard1
17:51:08  < t:2024-11-14 15:51:05,916 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_small7.standard1
18:01:11  < t:2024-11-14 16:01:08,301 f:full_storage_utilization_test_2.py l:25   c:FullStorageUtilizationTest2 p:INFO  > Executing DROP TABLE keyspace_small5.standard1

Disk usage drops to about 56%

---

Usage % and P99 latencies

Usage in GB

Argus

Argus link https://argus.scylladb.com/tests/scylla-cluster-tests/00e90f4b-46ae-489d-b3a4-12b42e540a5e

[cezarmoise]

Updated https://github.com/scylladb/scylla-cluster-tests/issues/9166#issuecomment-2477207563 with better graph.