search

DynamoRIO / dynamorio

Dynamic Instrumentation Tool Platform
Other
2.61k stars 554 forks source link

TEST REPORTING BUG: Wrong issue number can be reported for failing tests that are ignored #6607

Open xdje42 opened 7 months ago

xdje42 commented 7 months ago

Describe the bug

Test runs can report the wrong issue number for ignored tests.

To Reproduce

See for example, https://github.com/DynamoRIO/dynamorio/actions/runs/7731301531/job/21078628227#step:7:84492

====> FAILURE in debug-internal-64 <====
debug-internal-64: 902 tests passed, **** 6 tests failed, of which 2 were flaky, but ignoring 3 for i#2941: ****
    (ignore: i#2941)    code_api|client.attach_test 
    code_api|api.static_maps_mixup_novars_FLAKY 
    (ignore: i#2941)    code_api,opt_memory|common.loglevel 
    (ignore: i#2941)    code_api,opt_speed|common.decode-stress 
    code_api|tool.drcachesim.invariants 
    code_api|api.static_sideline_FLAKY 
WARNING: maximum warning/error limit hit for release-external-64!
  Manually verify whether it succeeded.

Note i#2941 is being reported as the issue for the attach_test failure. We'll need to augment ignored test marking to let one specify the issue so that it makes it through to the test output.