[test-triage] rv_dm_smoke is very flaky, DV_CHECK_EQ(rwdata, 0) fails

lowRISC / opentitan

OpenTitan: Open source silicon root of trust

Apache License 2.0

2.58k stars 777 forks source link

Hierarchy of regression failure

Block level

Failure Description

UVM_INFO @ 0 ps: reporter [RNTST] Running test rv_dm_base_test...
UVM_INFO @         0 ps: (rv_dm_env_cfg.sv:45) [cfg] ral_model_names: '{"rv_dm_regs_reg_block", "rv_dm_mem_reg_block"}
UVM_INFO @         0 ps: (rv_dm_env_cfg.sv:67) [cfg] Fixing reset values in jtag_dmi_ral
UVM_INFO @         0 ps: (sba_access_monitor.sv:62) uvm_test_top.env.m_sba_access_monitor [uvm_test_top.env.m_sba_access_monitor] sba_addrs: '{'h38, 'h39, 'h3c}
UVM_ERROR @  21823314 ps: (rv_dm_scoreboard.sv:83) [uvm_test_top.env.scoreboard] Check failed rwdata == 0 (1 [0x1] vs 0 [0x0])
UVM_INFO @  21823314 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]

From hw/ip/rv_dm/dv/env/rv_dm_scoreboard.sv, the DV_CHECK_EQ that fails:

      case (selected_dtm_csr.get_name())
        "bypass0", "bypass1": begin
          // TDI gets shifted out of TDO, and appears left-shifted by 1 - new write is ignored.
          uvm_reg_data_t rwdata;
          rwdata = get_field_val(selected_dtm_csr.get_field_by_name("bypass"), item.dout);
          `DV_CHECK_EQ(rwdata, 0)
          void'(selected_dtm_csr.predict(.value(item.dr), .kind(UVM_PREDICT_WRITE)));
        end

Steps to Reproduce

GitHub Revision: c4666f429474acc507064e5448fa4d347fe5e611
dvsim invocation command to reproduce the failure: ./util/dvsim/dvsim.py hw/ip/rv_dm/dv/rv_dm_sim_cfg.hjson -i rv_dm_smoke --build-seed 85829748320245376283659198434338498577935164172956485671224275001047693479661 --fixed-seed 62867758507077141611937936609975855929626662239195528877700989414354816228375

It is very easy to find other failing seeds, so this test has a high level of flakiness.

Tests with similar or related failures

rv_dm_jtag_dtm_idle_hint fails with the same error with the command ./util/dvsim/dvsim.py hw/ip/rv_dm/dv/rv_dm_sim_cfg.hjson -i rv_dm_jtag_dtm_idle_hint --build-seed 85829748320245376283659198434338498577935164172956485671224275001047693479661. Other failing seeds, for the same commit:

21398480815814838971835787872663989613314371603139767217588345495592569462190
15717867802553057123626730855244185211380337494610388407941957578997623508128

## RV_DM Simulation Results ### Monday June 03 2024 10:33:31 UTC ### GitHub Revision: [`4facfc9721`](https://github.com/lowrisc/opentitan/tree/4facfc972126b64a4032c875095f7feb58f0e0c3) ### Branch: tmp ### [Testplan](https://opentitan.org/book/hw/ip/rv_dm/data/rv_dm_testplan.html) ### Simulator: XCELIUM ### Test Results | Stage | Name | Tests | Max Job Runtime | Simulated Time | Passing | Total | Pass Rate | |:-------:|:------:|:------------|:-----------------:|:----------------:|:---------:|:-------:|:-----------:| | V1 | smoke | rv_dm_smoke | 27.000s | 4.790ms | 50 | 50 | 100.00 % | | V1 | | **TOTAL** | | | 50 | 50 | 100.00 % | | V2 | idcode | rv_dm_smoke | 27.000s | 4.790ms | 50 | 50 | 100.00 % | | | | **TOTAL** | | | 50 | 50 | 100.00 % |

## RV_DM Simulation Results ### Monday June 03 2024 10:35:56 UTC ### GitHub Revision: [`4facfc9721`](https://github.com/lowrisc/opentitan/tree/4facfc972126b64a4032c875095f7feb58f0e0c3) ### Branch: tmp ### [Testplan](https://opentitan.org/book/hw/ip/rv_dm/data/rv_dm_testplan.html) ### Simulator: XCELIUM ### Test Results | Stage | Name | Tests | Max Job Runtime | Simulated Time | Passing | Total | Pass Rate | |:-------:|:------------------:|:-------------------------|:-----------------:|:----------------:|:---------:|:-------:|:-----------:| | V2 | jtag_dtm_idle_hint | rv_dm_jtag_dtm_idle_hint | 6.000s | 521.911us | 50 | 50 | 100.00 % | | V2 | | **TOTAL** | | | 50 | 50 | 100.00 % | | | | **TOTAL** | | | 50 | 50 | 100.00 % |

lowRISC / opentitan