search

valkey-io / valkey

A flexible distributed key-value datastore that supports both caching and beyond caching workloads.
https://valkey.io
Other
16.06k stars 599 forks source link

[Test issue] The tests don't correctly reflect the number of tests left #1017

Open madolson opened 1 week ago

madolson commented 1 week ago

It repeatedly says we are at 124/124 when running, https://github.com/valkey-io/valkey/actions/runs/10805625870/job/29973022674?pr=1016#step:4:5651. I guess you can replicate this with just make test. Not sure what is causing it.

[124/124 done]: integration/replication-psync (324 seconds)
Testing solo test
=== () Starting server 127.0.0.1:28738 ok
[ignore]: XADD one huge field: large memory flag not provided
=== () Starting server 127.0.0.1:28739 ok
[ignore]: XADD one huge field - 1: large memory flag not provided
=== () Starting server 127.0.0.1:28740 ok
[ignore]: several XADD big fields: large memory flag not provided
=== () Starting server 127.0.0.1:28741 ok
[ignore]: single XADD big fields: large memory flag not provided
=== () Starting server 127.0.0.1:28742 ok
[ignore]: hash with many big fields: large memory flag not provided
=== () Starting server 127.0.0.1:28743 ok
[ignore]: hash with one huge field: large memory flag not provided
[124/124 done]: violations (1 seconds)
Testing solo test
=== (bitops) Starting server 127.0.0.1:28744 ok
[ignore]: BIT pos larger than UINT_MAX: large memory flag not provided
[ignore]: SETBIT values larger than UINT32_MAX and lzf_compress/lzf_decompress correctly: large memory flag not provided
[124/124 done]: bitops-large-memory (1 seconds)
hwware commented 1 week ago

I will let Shiv to check this (I do not know why I can not find his name here )

madolson commented 1 week ago

I will let Shiv to check this (I do not know why I can not find his name here )

He needs to either be an collaborator on the project, part of the valkey org, or needs to comment directly here. I'll invite him to the org.

Shivshankar-Reddy commented 1 week ago

I checked issue, found that the 124 is repeated for the solo test, which was grouped to test large-memory tests and they are not test files though they have name for the group. Seems the test files would be given number since these are already part of existing files count is not incremented. I also verified a old branch of valkey, redis as well and found that the behaviour is same and no consistency issue with number of tests.

Here is the link for the commit where they were grouped. https://github.com/valkey-io/valkey/commit//bd823c7fa3bce983be79835f9a306d4d7c15e31e

results on old branch of Valkey:

image

results on redis:

image

madolson commented 1 day ago

I think we can fix this by just removing the solo tests from the "completed tests" here: https://github.com/valkey-io/valkey/blob/unstable/tests/test_helper.tcl#L372.