ocaml-multicore / multicoretests

PBT testsuite and libraries for testing multicore OCaml
https://ocaml-multicore.github.io/multicoretests/
BSD 2-Clause "Simplified" License
37 stars 16 forks source link

Strengthen mutable_set_v5 expect test against timing differences #415

Closed jmid closed 11 months ago

jmid commented 11 months ago

The internal STM test mutable_set_v5.ml is currently run with --verbose which includes timing (0.0s) and subsequently diffed as an expect test. This creates the possibility of test failures due to a timing difference if it is run on a particular slow (CI) machine.

I just had this happen to me on the CI of an unrelated branch which happened to take 0.1s instead: https://github.com/ocaml-multicore/multicoretests/actions/runs/6946194440/job/18897107446

STM test exception during next_state sequential failed with Random_next_state_failure as expected
STM test exception during next_state parallel failed with Random_next_state_failure as expected
diff --git a/_build/default/test/mutable_set_v5.expected b/_build/default/test/mutable_set_v5.output
old mode 100755
new mode 100644
index 84ab110..ef19a64
--- a/_build/default/test/mutable_set_v5.expected
File "test/mutable_set_v5.expected", line 1, characters 0-0:
/usr/bin/git --no-pager diff --no-index --color=always -u _build/default/test/mutable_set_v5.expected _build/default/test/mutable_set_v5.output
Command exited with code 1.
File "test/mutable_set_v5.expected", line 1, characters 0-0:
/usr/bin/git --no-pager diff --no-index --color=always -u _build/default/test/mutable_set_v5.expected _build/default/test/mutable_set_v5.output
Command exited with code 1.
+++ b/_build/default/test/mutable_set_v5.output
@@ -1,6 +1,6 @@

random seed: 229109553
 generated error fail pass / total     time test name
-
[ ]    0    0    0    0 /  100     0.0s STM sequential tests
[ ]    0    0    0    0 /  100     0.0s STM sequential tests (generating)
[✗]    2    0    1    1 /  100     0.0s STM sequential tests
+
[ ]    0    0    0    0 /  100     0.0s STM sequential tests
[ ]    0    0    0    0 /  100     0.0s STM sequential tests (generating)
[✗]    2    0    1    1 /  100     0.1s STM sequential tests

 --- Failure --------------------------------------------------------------------

Error: Process completed with exit code 1.

This little PR therefore removes the --verbose option for this expect test.

jmid commented 11 months ago

CI summary for 790cc71:

Out of 63 workflows, 28 succeeded, 34 were canceled, 1 failed with a genuine failure

jmid commented 11 months ago

CI summary for 73d172c:

Out of 63 workflows, 58 succeeded, 2 were cancelled, 3 failed with 1 genuine error, 1 false alarm, 1 unknown (CI flaky)

jmid commented 11 months ago

CI summary for merge to main:

Out of 38 workflows, 2 failed with 1 borderline testsuite/ocaml5 issue, and 1 CI network issue