[test-triage] flaky rv_dm_stress_all test

nbdd0121 commented 1 month ago

Hierarchy of regression failure

Block level

Failure Description

UVM_ERROR (jtag_dmi_monitor.sv:112) m_jtag_dmi_monitor [m_jtag_dmi_monitor] Non-ok response seen with no previous DMI request. has 6 failures:

Test rv_dm_stress_all has 6 failures.

    2.rv_dm_stress_all.89804343375999490448275153506328839990367017626589673526725059932773130879573
    Line 255, in log /container/opentitan-public/scratch/os_regression/rv_dm-sim-vcs/2.rv_dm_stress_all/latest/run.log

      UVM_ERROR @ 1657183224 ps: (jtag_dmi_monitor.sv:112) uvm_test_top.env.m_jtag_dmi_monitor [uvm_test_top.env.m_jtag_dmi_monitor] Non-ok response seen with no previous DMI request.
      UVM_INFO @ 1657183224 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
      --- UVM Report catcher Summary ---

    13.rv_dm_stress_all.15294798192835239091549593238610712640703574634748948078112043569861655224499
    Line 255, in log /container/opentitan-public/scratch/os_regression/rv_dm-sim-vcs/13.rv_dm_stress_all/latest/run.log

      UVM_ERROR @ 947262753 ps: (jtag_dmi_monitor.sv:112) uvm_test_top.env.m_jtag_dmi_monitor [uvm_test_top.env.m_jtag_dmi_monitor] Non-ok response seen with no previous DMI request.
      UVM_INFO @ 947262753 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
      --- UVM Report catcher Summary ---

    ... and 4 more failures.

Steps to Reproduce

GitHub Revision: aad711e1bc33a541503e9a7a5e006ec87bd7602b, also reproduced with f02c6d9a89ec4d8f7d86d1856faabfcbc97bb38b
dvsim invocation command to reproduce the failure, inclusive of build and run seeds: util/dvsim/dvsim.py hw/ip/rv_dm/dv/rv_dm_sim_cfg.hjson -i rv_dm_stress_all --build-seed 38297581588405771015102977205739484163575733999253050480227719508149669635625 --fixed-seed 89804343375999490448275153506328839990367017626589673526725059932773130879573 --verbose

Tests with similar or related failures

No response

rswarbrick commented 1 month ago

On the plus side, this should be fixed soon. With three commits cherry-picked from three one-commit PRs, I just ran 32 seeds of rv_dm_stress_all and all of them passed. The PRs are:

24043
24095
24090

rswarbrick commented 1 month ago

In hindsight, the second commit on that list is definitely not going to be relevant. The third is merged, so I'm hopeful that this will be fixed when #24043 is merged (although we should probably wait for an overnight test to be sure)

andreaskurth commented 1 month ago

rv_dm_stress_all is back to 100% pass rate in the latest 4 nightlies, so I'm closing this issue as completed.

lowRISC / opentitan