search

mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.33k stars 1.53k forks source link

meson test --repeat doesn't run the expected number of tests #13194

Open DaanDeMeyer opened 3 weeks ago

DaanDeMeyer commented 3 weeks ago

Describe the bug

meson --repeat 1000 --num-processes 50 <test> only runs 163 times.

To Reproduce

~/systemd (main)> SYSTEMD_INTEGRATION_TESTS=1 TEST_MATCH_SUBTEST=".*cryptsetup.*" ../meson/meson.py test -C build --no-rebuild --repeat 1000 --num-processes 50 TEST-70-TPM2
   1/1000 systemd:integration-tests / TEST-70-TPM2           OK               55.48s
   2/1000 systemd:integration-tests / TEST-70-TPM2           OK               56.45s
   3/1000 systemd:integration-tests / TEST-70-TPM2           OK               57.80s
   4/1000 systemd:integration-tests / TEST-70-TPM2           OK               57.81s
   5/1000 systemd:integration-tests / TEST-70-TPM2           OK               59.01s
   6/1000 systemd:integration-tests / TEST-70-TPM2           OK               59.28s
   7/1000 systemd:integration-tests / TEST-70-TPM2           OK               61.07s
   8/1000 systemd:integration-tests / TEST-70-TPM2           OK               61.16s
   9/1000 systemd:integration-tests / TEST-70-TPM2           OK               61.93s
  10/1000 systemd:integration-tests / TEST-70-TPM2           OK               62.05s
  11/1000 systemd:integration-tests / TEST-70-TPM2           OK               62.43s
  12/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.06s
  13/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.10s
  14/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.48s
  15/1000 systemd:integration-tests / TEST-70-TPM2           OK               65.94s
  16/1000 systemd:integration-tests / TEST-70-TPM2           OK               66.86s
  17/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.04s
  18/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.72s
  19/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.01s
  20/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.71s
  21/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.86s
  22/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.02s
  23/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.75s
  24/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.92s
  25/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.69s
  26/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.66s
  27/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.86s
  28/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.22s
  29/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.83s
  30/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.16s
  31/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.62s
  32/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.10s
  33/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.51s
  34/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.95s
  35/1000 systemd:integration-tests / TEST-70-TPM2           OK               80.27s
  36/1000 systemd:integration-tests / TEST-70-TPM2           OK               81.76s
  37/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.39s
  38/1000 systemd:integration-tests / TEST-70-TPM2           OK               86.86s
  39/1000 systemd:integration-tests / TEST-70-TPM2           OK               88.66s
  40/1000 systemd:integration-tests / TEST-70-TPM2           OK               89.20s
  41/1000 systemd:integration-tests / TEST-70-TPM2           OK               91.51s
  42/1000 systemd:integration-tests / TEST-70-TPM2           OK               92.66s
  43/1000 systemd:integration-tests / TEST-70-TPM2           OK               95.16s
  44/1000 systemd:integration-tests / TEST-70-TPM2           OK               95.34s
  45/1000 systemd:integration-tests / TEST-70-TPM2           OK               98.02s
  46/1000 systemd:integration-tests / TEST-70-TPM2           OK              102.47s
  47/1000 systemd:integration-tests / TEST-70-TPM2           OK              105.15s
  48/1000 systemd:integration-tests / TEST-70-TPM2           OK              106.65s
  49/1000 systemd:integration-tests / TEST-70-TPM2           OK              108.77s
  50/1000 systemd:integration-tests / TEST-70-TPM2           OK              111.83s
  51/1000 systemd:integration-tests / TEST-70-TPM2           OK               62.02s
  52/1000 systemd:integration-tests / TEST-70-TPM2           OK               63.04s
  53/1000 systemd:integration-tests / TEST-70-TPM2           OK               64.32s
  54/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.03s
  55/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.07s
  56/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.44s
  57/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.07s
  58/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.34s
  59/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.75s
  60/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.50s
  61/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.29s
  62/1000 systemd:integration-tests / TEST-70-TPM2           OK               66.87s
  63/1000 systemd:integration-tests / TEST-70-TPM2           OK               67.86s
  64/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.18s
  65/1000 systemd:integration-tests / TEST-70-TPM2           OK               68.99s
  66/1000 systemd:integration-tests / TEST-70-TPM2           OK               69.30s
  67/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.28s
  68/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.51s
  69/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.02s
  70/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.00s
  71/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.35s
  72/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.45s
  73/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.85s
  74/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.34s
  75/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.54s
  76/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.58s
  77/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.52s
  78/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.55s
  79/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.24s
  80/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.18s
  81/1000 systemd:integration-tests / TEST-70-TPM2           OK               75.33s
  82/1000 systemd:integration-tests / TEST-70-TPM2           OK               74.60s
  83/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.53s
  84/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.31s
  85/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.04s
  86/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.03s
  87/1000 systemd:integration-tests / TEST-70-TPM2           OK               79.78s
  88/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.85s
  89/1000 systemd:integration-tests / TEST-70-TPM2           OK               80.62s
  90/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.86s
  91/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.87s
  92/1000 systemd:integration-tests / TEST-70-TPM2           OK               84.56s
  93/1000 systemd:integration-tests / TEST-70-TPM2           OK               82.47s
  94/1000 systemd:integration-tests / TEST-70-TPM2           OK               83.63s
  95/1000 systemd:integration-tests / TEST-70-TPM2           OK               81.07s
  96/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.64s
  97/1000 systemd:integration-tests / TEST-70-TPM2           OK               78.29s
  98/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.73s
  99/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.72s
 100/1000 systemd:integration-tests / TEST-70-TPM2           OK               73.95s
 101/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.09s
 102/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.87s
 103/1000 systemd:integration-tests / TEST-70-TPM2           OK               70.48s
 104/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.26s
 105/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.18s
 106/1000 systemd:integration-tests / TEST-70-TPM2           OK               72.43s
 107/1000 systemd:integration-tests / TEST-70-TPM2           OK               71.75s
 108/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.41s
 109/1000 systemd:integration-tests / TEST-70-TPM2           OK               76.83s
 110/1000 systemd:integration-tests / TEST-70-TPM2           OK               77.47s
 111/1000 systemd:integration-tests / TEST-70-TPM2           OK               79.79s
 112/1000 systemd:integration-tests / TEST-70-TPM2           OK               84.95s
 113/1000 systemd:integration-tests / TEST-70-TPM2           OK               84.18s
 114/1000 systemd:integration-tests / TEST-70-TPM2           OK               93.75s
 115/1000 systemd:integration-tests / TEST-70-TPM2           OK               95.21s
 116/1000 systemd:integration-tests / TEST-70-TPM2           FAIL             60.88s   exit status 1
>>> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=139 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=191 /home/daandemeyer/systemd/test/integration-test-wrapper.py --meson-source-dir /home/daandemeyer/systemd --meson-build-dir /home/daandemeyer/systemd/build --test-name TEST-70-TPM2 --test-number 70 --storage volatile --firmware linux --

 117/1000 systemd:integration-tests / TEST-70-TPM2           FAIL             59.62s   exit status 1
>>> UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=13 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MESON_TEST_ITERATION=141 /home/daandemeyer/systemd/test/integration-test-wrapper.py --meson-source-dir /home/daandemeyer/systemd --meson-build-dir /home/daandemeyer/systemd/build --test-name TEST-70-TPM2 --test-number 70 --storage volatile --firmware linux --

 118/1000 systemd:integration-tests / TEST-70-TPM2           OK               98.82s
 119/1000 systemd:integration-tests / TEST-70-TPM2           OK               96.76s
 120/1000 systemd:integration-tests / TEST-70-TPM2           OK              100.11s
 121/1000 systemd:integration-tests / TEST-70-TPM2           OK               97.17s
 122/1000 systemd:integration-tests / TEST-70-TPM2           OK               99.31s
 123/1000 systemd:integration-tests / TEST-70-TPM2           OK               98.12s
 124/1000 systemd:integration-tests / TEST-70-TPM2           OK               99.14s
 125/1000 systemd:integration-tests / TEST-70-TPM2           OK              103.14s
 126/1000 systemd:integration-tests / TEST-70-TPM2           OK              110.80s
 127/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.31s
 128/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.84s
 129/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.26s
 130/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.58s
 131/1000 systemd:integration-tests / TEST-70-TPM2           OK              127.54s
 132/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.91s
 133/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.60s
 134/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.41s
 135/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.01s
 136/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.63s
 137/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.43s
 138/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.16s
 139/1000 systemd:integration-tests / TEST-70-TPM2           OK              127.08s
 140/1000 systemd:integration-tests / TEST-70-TPM2           OK              123.76s
 141/1000 systemd:integration-tests / TEST-70-TPM2           OK              115.70s
 142/1000 systemd:integration-tests / TEST-70-TPM2           OK              117.06s
 143/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.16s
 144/1000 systemd:integration-tests / TEST-70-TPM2           OK              116.15s
 145/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.43s
 146/1000 systemd:integration-tests / TEST-70-TPM2           OK              118.52s
 147/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.52s
 148/1000 systemd:integration-tests / TEST-70-TPM2           OK              123.13s
 149/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.71s
 150/1000 systemd:integration-tests / TEST-70-TPM2           OK              124.56s
 151/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.38s
 152/1000 systemd:integration-tests / TEST-70-TPM2           OK              125.55s
 153/1000 systemd:integration-tests / TEST-70-TPM2           OK              123.43s
 154/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.78s
 155/1000 systemd:integration-tests / TEST-70-TPM2           OK              122.82s
 156/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.56s
 157/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.95s
q 158/1000 systemd:integration-tests / TEST-70-TPM2           OK              120.10s
 159/1000 systemd:integration-tests / TEST-70-TPM2           OK              121.37s
 160/1000 systemd:integration-tests / TEST-70-TPM2           OK              127.26s
 161/1000 systemd:integration-tests / TEST-70-TPM2           OK              120.02s
 162/1000 systemd:integration-tests / TEST-70-TPM2           OK              126.15s
 163/1000 systemd:integration-tests / TEST-70-TPM2           OK              114.98s
 164/1000 systemd:integration-tests / TEST-70-TPM2           OK              114.13s
 165/1000 systemd:integration-tests / TEST-70-TPM2           OK              130.51s

Ok:                 163 
Expected Fail:      0   
Fail:               2   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/daandemeyer/systemd/build/meson-logs/testlog.txt

Expected behavior

The test runs 1000 times.

system parameters