dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.96k stars 954 forks source link

fix: slow regression tests tests #4117

Closed kostasrim closed 1 week ago

kostasrim commented 1 week ago

We have a few very slow tests on the CI. Specifically:

241.30s call     dragonfly/replication_test.py::test_replication_all[df_factory0-mode0-8-t_replicas8-seeder_config8-50000-True]
240.80s call     dragonfly/replication_test.py::test_replication_all[df_factory0-mode0-8-t_replicas7-seeder_config7-50000-False]
239.92s call     dragonfly/replication_test.py::test_replication_all[df_factory0-mode1-8-t_replicas7-seeder_config7-50000-False]
236.87s call     dragonfly/replication_test.py::test_replication_all[df_factory0-mode1-8-t_replicas8-seeder_config8-50000-True]
178.18s call     dragonfly/snapshot_test.py::test_big_value_serialization_memory_limit[HSET-df_factory0]
175.97s call     dragonfly/snapshot_test.py::test_big_value_serialization_memory_limit[SADD-df_factory0]
175.79s call     dragonfly/snapshot_test.py::test_big_value_serialization_memory_limit[ZSET-df_factory0]
172.35s call     dragonfly/snapshot_test.py::test_big_value_serialization_memory_limit[LIST-df_factory0]
60.21s call     dragonfly/connection_test.py::test_pubsub_busy_connections[df_factory0]

This PR changes:

It refactors the test_big_value_serialization. First, the test was incorrect, in fact for some reason it did not even run with big value serialization flag set. Second, the test used execute_command in a loop which took a substantial amount of time. The changes on my machine reduce the total running time of the test from 4 * 170 seconds on average to a staggering 1 minute and 10 seconds for all 4.

Also notice, that we don't really need to test big value serialization on a stress test. It's redundant because when we set the value to a small number (4096 etc) it will have the same effect regardless of the stress load. I removed that one test case, saving as roughly (235 * 2) seconds

kostasrim commented 1 week ago

Impact on a full run: https://github.com/dragonflydb/dragonfly/actions/runs/11796502461

(I will post the results once it completes)

kostasrim commented 1 week ago

x86 debug 28 minutes now takes 22 arm debug 42 minutes now takes 34 x86 release 28 minutes now takes 19 arm release 46 minutes now takes 28

Savings on a full run 24minutes (because arm tests run sequentially)

https://github.com/dragonflydb/dragonfly/actions/runs/11797723885/job/32862315798 vs https://github.com/dragonflydb/dragonfly/actions/runs/11793906726/job/32850430451

kostasrim commented 1 week ago

At some point I will take care of:

180.20s call dragonfly/replication_test.py::test_replicaof_reject_on_load[df_seeder_factory0-df_factory0]

It's the next low hanging fruit