riscv gdb remote: breakpoint cannot be inserted

Use QEMU v5.2.0, lauch a guest by:

$ qemu-system-riscv64 -display none -serial stdio -M virt -smp 4 -m 2G -kernel vxWorks -trace gdbstub* -D gdbstub.log -s -S

Use gdb v10.1 to do remote debugging:

>>> target remote :1234
Remote debugging using :1234
0x0000000000001000 in ?? ()
>>> b usrRoot
Cannot access memory at address 0xffffffff80202cac
>>> b vxbXspiFlashAttach
Breakpoint 1 at 0xffffffff80309852: file vxbXspiFlash.c, line 525.
>>> b usrRoot
Cannot access memory at address 0xffffffff80202cac
>>> c
Continuing.

Examine the gdbstub.log file, we see when inserting breakpoint at usrRoot, no "Z0" packet has been sent to QEMU gdbstub.

18590@1609688046.723671:gdbstub_io_command Received: mffffffff80202c80,40
18590@1609688046.723720:gdbstub_io_reply Sent: E14
18590@1609688046.723788:gdbstub_io_got_ack Got ACK
18590@1609688046.723819:gdbstub_io_command Received: mffffffff80202cac,1
18590@1609688046.723826:gdbstub_io_reply Sent: E14
18590@1609688046.723885:gdbstub_io_got_ack Got ACK
18590@1609688046.723914:gdbstub_io_command Received: mffffffff80202c80,40
18590@1609688046.723921:gdbstub_io_reply Sent: E14
18590@1609688046.724005:gdbstub_io_got_ack Got ACK
18590@1609688046.724035:gdbstub_io_command Received: mffffffff80202cac,1
18590@1609688046.724042:gdbstub_io_reply Sent: E14
18590@1609688046.724089:gdbstub_io_got_ack Got ACK
18590@1609688058.136639:gdbstub_io_command Received: mffffffff80309852,2
18590@1609688058.136680:gdbstub_io_reply Sent: E14
18590@1609688058.136727:gdbstub_io_got_ack Got ACK
18590@1609688060.127187:gdbstub_io_command Received: mffffffff80202c80,40
18590@1609688060.127230:gdbstub_io_reply Sent: E14
18590@1609688060.127285:gdbstub_io_got_ack Got ACK
18590@1609688060.127322:gdbstub_io_command Received: mffffffff80202cac,1
18590@1609688060.127328:gdbstub_io_reply Sent: E14
18590@1609688060.127379:gdbstub_io_got_ack Got ACK
18590@1609688060.127404:gdbstub_io_command Received: mffffffff80202c80,40
18590@1609688060.127409:gdbstub_io_reply Sent: E14
18590@1609688060.127437:gdbstub_io_got_ack Got ACK
18590@1609688060.127459:gdbstub_io_command Received: mffffffff80202cac,1
18590@1609688060.127463:gdbstub_io_reply Sent: E14
18590@1609688060.127487:gdbstub_io_got_ack Got ACK
18590@1609688202.154920:gdbstub_io_command Received: Z0,ffffffff80309852,2
18590@1609688202.155062:gdbstub_io_reply Sent: OK
18590@1609688202.155138:gdbstub_io_got_ack Got ACK
18590@1609688202.155200:gdbstub_io_command Received: vCont?
18590@1609688202.155207:gdbstub_io_reply Sent: vCont;c;C;s;S
18590@1609688202.155243:gdbstub_io_got_ack Got ACK
18590@1609688202.155312:gdbstub_io_command Received: vCont;c:p1.-1
18590@1609688202.155367:gdbstub_op_continue_cpu Continuing CPU 0
18590@1609688202.155371:gdbstub_op_continue_cpu Continuing CPU 1
18590@1609688202.155374:gdbstub_op_continue_cpu Continuing CPU 2
18590@1609688202.155377:gdbstub_op_continue_cpu Continuing CPU 3

It seems to me that the gdb client has never requested the QEMU gdbstub to insert the breakpoint at 0xffffffff80202cac. Only the 0xffffffff80309852 has been inserted a breakpoint (Z0,ffffffff80309852,2).

I suspect there is something wrong on the gdb side. Any idea what could be wrong?

We don't have a gdb guy. You might be better off just sending bug reports to the FSF GDB mailing lists.

By default, the RISC-V gdb port doesn't know if compressed breakpoints are safe, so it needs to read the target memory first, check to see if it is a compressed or non-compressed instruction, and then emit a compressed or non-compressed breakpoint as appropriate. But unaligned data reads are not supported by some hardware, so there is a short-cut where if the address is 2-byte aligned then we skip the memory read and just write a compressed breakpoint. The working address above is 2-byte aligned, and it looks like memory reads of code are failing. If we can't read code to see if it is a compressed or not instruction at the target, then we can't emit a breakpoint. But we will if it is 2-byte aligned.

There is a setting for breakpoint type. By default it is set to auto. (gdb) show riscv use-compressed-breakpoints Debugger's use of compressed breakpoints is set to auto. In auto mode we need to read memory to see if it is a compressed instruction or not. If you know that compressed breakpoints are safe, then you can set this to on (gdb) set riscv use-compressed-breakpoints on and now we won't try to read memory first. We will just always write a compressed breakpoint.

Thanks Jim. The workaround works.

>>> target remote :1234
Remote debugging using :1234
0x0000000000001000 in ?? ()
>>> show riscv use-compressed-breakpoints
Debugger's use of compressed breakpoints is set to auto.
>>> set riscv use-compressed-breakpoints on
>>> b usrRoot
Breakpoint 1 at 0xffffffff80202cac: file ../prjConfig.c, line 812.
>>> b vxbXspiFlashAttach
Breakpoint 2 at 0xffffffff80309852: file vxbXspiFlash.c, line 525.
>>> c
Continuing.

I have some questions:

By default, the RISC-V gdb port doesn't know if compressed breakpoints are safe

Does this mean RISC-V gdb cannot know if target supports C extension? How about automatically set the compressed breakpoint when seeting ELF flags RVC in the ELF header?

so there is a short-cut where if the address is 2-byte aligned

So the short-cut condition is that the address is 2-byte aligned, but not 4-byte aligned?

Another strange thing is when I was debugging QEMU gdbstub I let it stop at a breakpoint gdb_handle_packet(), when "b usrRoot" was received from the GDB client, after several stepping in and continue, the breakpoint can be set correctly.

Thanks Jim. The workaround works.

Perhaps there is an issue with the qemu gdb stub, since gdb should be able to read the memory if it can write a breakpoint to it.

By default, the RISC-V gdb port doesn't know if compressed breakpoints are safe

Does this mean RISC-V gdb cannot know if target supports C extension? How about automatically set the compressed breakpoint when seeting ELF flags RVC in the ELF header?

You can connect gdb to a target without providing an ELF binary first, so we can't always rely on the existence of a binary.

There are also more subtle issues. For instance, a binary could have run -time checks for hardware support for C instructions, and only use the C support on targets that have it. In which case a binary with the Elf header C bit set might be running on non-C hardware where a compressed breakpoint won't work. Or on a linux system, you could have a non-C application linked against a C shared library, so using the application C bit could result in corruption of the shared library if you write a 4-byte breakpoint over top a 2-byte compressed instruction. We would then have to track multiple ELF binaries and shared libraries and switch breakpoint mode depending on which object we are currently inside. And this assumes we can access the ELF headers for a shared library, I know that glibc supports this, but I don't know if all libraries support this.

Only writing a compressed breakpoint over top a compressed instruction is always safe, and simpler than other solutions.

so there is a short-cut where if the address is 2-byte aligned

So the short-cut condition is that the address is 2-byte aligned, but not 4-byte aligned?

Yes. An instruction can only be 2-byte but not 4-byte aligned if the target supports C. In this case, using a compressed breakpoint is always safe.

Another strange thing is when I was debugging QEMU gdbstub I let it stop at a breakpoint gdb_handle_packet(), when "b usrRoot" was received from the GDB client, after several stepping in and continue, the breakpoint can be set correctly.

Maybe this is related to the point I made at the top? Why can't gdb read the memory? Maybe the gdb stub is using the wrong memory access function to try to read memory, and getting the wrong permissions applied to its read attempt.

In which case a binary with the Elf header C bit set might be running on non-C hardware where a compressed breakpoint won't work.

I guess you wanted to say "a binary with the Elf header C bit NOT set might be running on C hardware"? Yep, I understand the situation of mixed C and non-C usage, either on the hardware or shared library.

To summerize, the current gdb solution:

Using "set riscv use-compressed-breakpoints on" to handle breakpoint insersion at a 4-byte aligned address when a memory read of the breakpoint address is not feasible, on a C hardware
Trying to read a memory of the 4-byte aligned breakpoint address to determine whether a C or non-C breakpoint needs to be inserted
For a 2-byte aligned breakpoint address, C breakpoint is always used without memory read

sounds correct to me and we don't have other options here. I am not sure "sending bug reports to the FSF GDB mailing lists" helps anything.

Regarding

Perhaps there is an issue with the qemu gdb stub, since gdb should be able to read the memory if it can write a breakpoint to it.

QEMU gdbstub does nothing wrong. It's just that the breakpoint address is a virtual address and at that point no MMU page table has been estabilished on the target, hence QEMU cannot read anything.

riscv-collab / riscv-gnu-toolchain

riscv gdb remote: breakpoint cannot be inserted #811