pamaury
commented
1 year ago Observations regarding rom_e2e_debug_test_otp_test_unlocked0
I tried to debug the problem: this tests fails consistently on fred. Step to get more debugging:
- add
debug_level 3toutil/openocd/target/lowrisc_earlgrey.cfg=> very verbose output - edit
rules/script/gdb_test_coordinator.pyto keep logging after GDB exits:# in __init__(): change this line to remove styling: self.console = rich.console.Console() # no more colors # in echo(), add: if line == "": return # add method def flush_all(self) -> None: now = time.time_ns() while time.time_ns() <= now + 1000000000: self.maybe_print_output(1) # in main: while not background.empty(): .... if returncode != 0: # wait a little bit so that we can flush the output of all processes background.flush_all() # <--- this is new sys.exit(returncode)Conclusion: test fails when reading from RAM (
x/16xb 0x10000000). OpenOCD output:Debug: 34789 6184 gdb_server.c:390 gdb_log_incoming_packet(): [riscv.tap.0] received packet: m10000000,1 Debug: 34790 6184 gdb_server.c:1524 gdb_read_memory_packet(): addr: 0x0000000010000000, len: 0x00000001 Debug: 34791 6184 target.c:2475 target_read_buffer(): reading buffer of 1 byte at 0x10000000 Debug: 34792 6184 riscv.c:3306 riscv_set_current_hartid(): setting hartid to 0, was 0 Debug: 34793 6184 riscv-013.c:4077 riscv013_get_register(): [riscv.tap.0] reading register priv Debug: 34794 6184 riscv.c:3306 riscv_set_current_hartid(): setting hartid to 0, was 0 Debug: 34795 6184 riscv-013.c:782 execute_abstract_command(): command=0x2207b0; access register, size=32, postexec=0, transfer=1, write=0, regno=0x7b0 .... 013.c:407 scan(): -> sberror=7 Debug: 34832 6191 riscv-013.c:2850 log_mem_access_result(): Failed to read memory via system bus. Error: 34833 6191 riscv-013.c:3598 read_memory(): Target riscv.tap.0: Failed to read memory (addr=0x10000000) Error: 34834 6191 riscv-013.c:3599 read_memory(): progbuf=disabled, sysbus=failed, abstract=disabled Debug: 34835 6191 gdb_server.c:1483 gdb_error(): Reporting -4 to GDB as generic errorRemark: reading a register works fine, for example:
echo ::: Read memory (LC_CTRL_STATE) x/4xb 0x40140034More observations: after setting
sp, GDB seems to want to read the stack all the time, but it fails becausespcontains a nonsense value. Also, reading the RAM before this step works, but fails after:[GDB] ::: Read memory (LC_CTRL_STATE) [GDB] 0x40140034: 0x21 0x84 0x10 0x02 [GDB] ::: Read RAM [GDB] 0x10000000 <boot_measurements>: 0x10 [GDB] :::: Read and write GPRs. [GDB] :::: Copy registers.::: Read RAM 2 [GDB] 0x10000000 <boot_measurements>: 0x10 [GDB] :::: Set registers. [GDB] :::: Set zero to 0xdeadbeef. [GDB] :::: Set ra to 0xdeadbeef. [GDB] :::: Set sp to 0xdeadbeef. [OPENOCD] Warn : Failed to read memory via system bus. [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf00) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf0b) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf00) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf0b) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf00) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf0b) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf00) [OPENOCD] Error: progbuf=disabled, sysbus=failed, abstract=disabled [OPENOCD] Error: Target riscv.tap.0: Failed to read memory (addr=0xdeadbf0b) [GDB] :::: Set gp to 0xdeadbeef.
timothytrippel
a-will
andreaskurth
engdoreis
johngt
msfschaffner
moidx
Hierarchy of regression failure
Chip Level
Failure Description
The ROM E2E tests that use the
rv_dm + openocd + gdbare shaky lately.The reason should be investigated, and the tests should be improved.
Steps to Reproduce
Tests with similar or related failures