[test-triage,rom_e2e,opentitanlib] gdb/openocd/jtag tests flaky/failing

lowRISC / opentitan

OpenTitan: Open source silicon root of trust

https://www.opentitan.org

Apache License 2.0

2.6k stars 783 forks source link

[test-triage,rom_e2e,opentitanlib] gdb/openocd/jtag tests flaky/failing #17729

Closed engdoreis closed 1 year ago

engdoreis commented 1 year ago

Hierarchy of regression failure

Chip Level

Failure Description

The ROM E2E tests that use the rv_dm + openocd + gdb are shaky lately.

The reason should be investigated, and the tests should be improved.

Steps to Reproduce

GitHub Revision: master
bazel test --define DISABLE_VERILATOR_BUILD=true --define bitstream=gcp_splice --test_tag_filters=-verilator,-dv,-broken --build_tests_only //sw/device/silicon_creator/rom/e2e:rom_e2e_debug_test_otp_test_unlocked0

Tests with similar or related failures

[ ] rom_e2e_debug_test_otp_test_unlocked0
[ ] debug_disallowed_in_prod_fpga_cw310_test_otp_rma
[ ] rom_e2e_asm_watchdog_bark_fpga_cw310_test_otp_rma
[ ] rom_e2e_asm_watchdog_bite_fpga_cw310_test_otp_test_unlocked0
[ ] rom_e2e_asm_watchdog_bite_fpga_cw310_test_otp_dev
[ ] sram_program_fpga_cw310_test_otp_test_unlocked0

pamaury commented 1 year ago

I think I have been to track down the issue: the CW310 backend always reconfigures the reset and strap pins on init. At the same time, the gdb coordinator script starts the opentitantool console after starting OpenOCD. This means that when it initializes the CW310 backend, this will mess up with the pins for a few milliseconds.

I have been able to confirm in tests that this is a failure mode that we have observed. I am currently running tests in CI to see if this solves the issue completely or not.

timothytrippel commented 1 year ago

Awesome work everyone! Since we do not use the gdb coordinator script for manufacturing (for the complexity reasons) this should not be an issue.

Long term, as a lot of us have discussed, we should remove the gdb coordinator script and the custom gdb/openocd Bazel test rule and move the GDB coordination into opentitantool, since we don't need it for writing custom tests now that we drive OpenOCD with opentitanlib directly.

timothytrippel commented 1 year ago

can we close this now? @pamaury @andreaskurth

timothytrippel commented 1 year ago

nevermind I see #18554 links to close this when merged. Thanks everyone!

a-will commented 1 year ago

After the freeze, I can also remove the POR connection to SRSTn on the ARM JTAG connectors. The POR functionality was documented, but when I hooked it up, I didn't anticipate the confusion that would cause, nor how it might get driven from tools that think it's ARM's SRSTn.

The commit for this is https://github.com/a-will/opentitan/commit/414cd10d113263843b55d4aedc76bd70c3108790, but I won't create a PR for that right now (to avoid the added load to CI).

msfschaffner commented 1 year ago

Thanks everyone for helping to root cause this!