ocaml-multicore / multicoretests

PBT testsuite and libraries for testing multicore OCaml
https://ocaml-multicore.github.io/multicoretests/
BSD 2-Clause "Simplified" License
37 stars 16 forks source link

Expand STM Semaphore tests #419

Closed jmid closed 10 months ago

jmid commented 10 months ago

While trying out a run of multicoretests under TSan, I spotted some limitations of our STM Semaphore tests. For one, we test Semaphore.Counting.get_value in parallel despite it being documented as racy.

This PR

Because of the precondition in the Semaphore.Binary STM test, it needs to generate extra inputs. By default, qcheck-stm tries to generate up to 3x as many inputs as needed which is not quite enough to hit 500:

$ dune exec src/semaphore/stm_tests_binary.exe -- -v
random seed: 338133583
generated error fail pass / total     time test name
[✓]  500    0    0  500 /  500     0.1s STM Semaphore.Binary test sequential
[✓] 1500    0    0  332 /  500     2.1s STM Semaphore.Binary test parallel
================================================================================
success (ran 2 tests)

The test is still relatively fast with a count of 500 compared to the STM Semaphore.Counting test which has a count of 200:

$ dune exec src/semaphore/stm_tests.exe -- -v
random seed: 72247306
generated error fail pass / total     time test name
[✓]  200    0    0  200 /  200     0.0s STM Semaphore.Counting test sequential
[✓]  455    0    0  200 /  200     2.1s STM Semaphore.Counting test parallel (w/ get_value)
[✓]  600    0    0  179 /  200     1.7s STM Semaphore.Counting test parallel (w/o get_value)
================================================================================
success (ran 3 tests)

As such I'm leaning towards declaring good enough...

jmid commented 10 months ago

CI summary:

Out of 63 workflows 3 failed with 1 genuine error and 2 false positives

jmid commented 10 months ago

CI summary for merge to main:

Out of 38 workflows 5 failed with 2 genuine issues, 2 CI issue, and 1 test suite false alarm.