search

DynamoRIO / dynamorio

Dynamic Instrumentation Tool Platform
Other
2.57k stars 551 forks source link

Thread is missing exit invariant failure on drcachesim.invariants test #6733

Open derekbruening opened 3 months ago

derekbruening commented 3 months ago

The x86_64 tool.drcachesim.invariants has failed on the master merge a number of times in the tool.drcachesim.invariants test:

Some older ones: https://github.com/DynamoRIO/dynamorio/actions/runs/7731301531/job/21078628227 https://github.com/DynamoRIO/dynamorio/actions/runs/7732983950/job/21084129568

A few more recent ones too. https://github.com/DynamoRIO/dynamorio/actions/runs/8442624559/job/23124442264

Trace invariant failure in T27434 at ref # 373883 (0 instrs since timestamp 13351202897274636): Thread is missing exit

Locally I can reproduce:

$ ctest --repeat-until-fail 500 -R sim.invariants

Failed after 29 successes.

edeiana commented 3 months ago

Also failed on AARCH64 ("aarch64-precommit (ubuntu-20-arm64-sve)") one time (so far) in #6691 (https://productionresultssa14.blob.core.windows.net/actions-results/7de917d3-3760-4c4e-8868-17a821dcbe85/workflow-job-run-aec6087b-1676-5d99-51e5-fc3628aaa01c/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-03-27T03%3A15%3A40Z&sig=4uo4KtcH6yeXG%2BApsmdkKeOQhQJNsAmoHkwgo4riuNE%3D&sp=r&spr=https&sr=b&st=2024-03-27T03%3A05%3A35Z&sv=2021-12-02).

edeiana commented 3 months ago

Sometimes it also times out (tested on x86_64):

$ ctest --repeat-until-fail 500 -R sim.invariants
Test project /OMITTED/dynamorio/build/dynamorio_inv_bug/build64
    Start 330: code_api|tool.drcachesim.invariants
    Test #330: code_api|tool.drcachesim.invariants ...***Timeout 180.02 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 180.10 sec

The following tests FAILED:
    330 - code_api|tool.drcachesim.invariants (Timeout)
Errors while running CTest
Output from these tests are in: /OMITTED/dynamorio/build/dynamorio_inv_bug/build64/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
derekbruening commented 3 months ago

This keeps failing periodically on post-merge runs, which we're trying to get green. E.g., failed just now: https://github.com/DynamoRIO/dynamorio/actions/runs/8470861218/job/23209561877