scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
57 stars 95 forks source link

Make `test_restore_benchmark` results comparable #9298

Open kreuzerkrieg opened 2 days ago

kreuzerkrieg commented 2 days ago

While we currently have a test for restore benchmarking, the configurations are set up in such a way that the results from different runs are not comparable. Ideally, restoring 1TB of data should take twice as long as restoring 0.5TB. However, due to the use of different instance types in each manager configuration, this expectation is not met. Additionally, ScyllaDB's autocompaction can interfere with the restore process, so the test should explicitly disable it to ensure more consistent and deterministic results.

Argus results 500GB run restore time [s] download bandwidth [MiB/s/shard] l&s bandwidth [MiB/s/shard] repair time [s] total [s]
500gb_1t_ics $${\color{red}00:57:09}$$ $${\color{green}17.95}$$ $${\color{green}7.46}$$ $${\color{red}00:18:06}$$ $${\color{red}01:15:15}$$
1TB run restore time [s] download bandwidth [MiB/s/shard] l&s bandwidth [MiB/s/shard] repair time [s] total [s]
1tb_1t_ics $${\color{red}02:07:50}$$ $${\color{green}17.46}$$ $${\color{green}6.48}$$ $${\color{red}00:43:28}$$ $${\color{red}02:51:18}$$

Also, additional results with 3 x i4i.4xlarge cluster

mikliapko commented 2 days ago

@kreuzerkrieg Could you please attach the run where your changes were tested?

kreuzerkrieg commented 2 days ago

@kreuzerkrieg Could you please attach the run where your changes were tested?

sure, will create new clean runs a add to the description shortly