"Register 3921 is not available" when debugging https://github.com/schoeberl/cae-examples on qemu

[zakkak]

https://github.com/schoeberl/cae-examples provides a bare-metal test for riscv.

Running it with:

qemu-system-riscv64 -nographic -S -s -kernel test

And stepping it through gdb with:

riscv64-unknown-elf-gdb test
target remote localhost:1234
ni
ni

Gives the message "Register 3921 is not available". Note that the test proceeds normally without any issues though.

[zakkak]

Looks related to https://github.com/riscv/riscv-gnu-toolchain/issues/217

[michaeljclark]

We've added CSRs to the GDB stub with an offset of 65. e.g.

if (reg >= 65) csrno = reg - 65

Register 834 is likely the misa CSR Register 3921 decodes to 0xF10 which is off-by-one from mvendorid

Can you test with this branch? https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v5

[mgrang]

I am using qemu in user mode. I just tried https://github.com/riscv/riscv-qemu/tree/qemu-upstream-v5 but I still see the error: Register 3921 is not available

Here are my steps: ` Compiled hello.c with -g. qemu-riscv32 -g 2222 hello.exe

riscv32-unknown-linux-gnu-gdb hello.exe target remote :2222 hit c

Register 3921 is not available (gdb) quit `

[jim-wilson]

0xf10 is misa in the v1.9 privilege spec. misa is 0x301 in the v1.9.1 and v1.10 privilege specs. There could be some priv spec confusion here.

However, looking at gdb, I see that it tries to read misa to determine whether there are FP registers or not, and for a few other reasons. The easiest way to trigger this is to set a breakpoint. It first tries to read 0x301, and if that fails, then it tries to read 0xf10 in case you have old hardware. The 0x301 read is failing silently, and then the 0xf10 read is failing with an error. I think this is because the first error gets caught, and the second error is not caught so the error is printed, which is a little confusing. The real problem is that the read of 0x301 failed. I think it failed because gdb tried to read the register from the current frame, instead of realizing that it is a special register that always must be read directly from the hardware. Anyways, this looks like more of a gdb problem than a qemu problem. And gdb is known to have a number of problems. There has been some gdb cleanup work done before it was posted as a patch upstream recently, and that work is not in the riscv-binutils-gdb tree, nor is it in the FSF gdb tree yet. The problem might already be fixed there.

[mgrang]

Thanks for the info Jim. If possible could you please point me to the GDB patch which may fix this issue. I can try to manually apply it to my RISCV GDB port and see if it resolved the issue?

[jim-wilson]

I don't know if the problem is fixed. I'm just pointing out that gdb sources are in flux at the moment because it is in the middle of being submitted upstream, and there won't be a proper stable gdb port until a patchset is accepted upstream.

You can see the v2 patchset here https://sourceware.org/ml/gdb-patches/2018-02/msg00409.html This is a patch against the upstream gdb sources, which currently have no RISC-V support. The upstream gdb web site is https://sourceware.org/gdb/

[zakkak]

I no longer (with current riscv-all branch) get the message following the above procedure, however if I try to set a breakpoint the message appears again.

[sandeepvijay]

I too get this issue while doing the step return. I am using GDB v8.0.50 and riscv qemu v2.12.50. Any one has found a solution yet?

Previously there was such an issue while putting a breakpoint - 'Register 834 is not available'. As you all pointed out, that was an issue with GDB and there was a workaround for it - 'set riscv use_compressed_breakpoint off'. The GDB version used then was 7.12.50. All the run time control commands were working with that version. Now I updated the RISCV GDB version to 8.0.50. With this version, I get the error 'Register 3921 is not available' while putting a breakpoint without setting the use_compressed_breakpoint to off. If I set it to off, then it works fine. I can put a breakpoint, run to the bp, step in, and step over. But then the step 'return' gives the error 'Register 3921 is not available'.

[lbmeng]

The issue is still reproducible with gdb/qemu built from the latest riscv/riscv-gnu-toolchain repo as of today. Basically the breakpoint does not work. Every time I set a breakpoint via 'b', gdb just complains: "Register 3921 is not available" and the breakpoint was not set at all.

[jim-wilson]

You need to specify how exactly you are using gdb, and whether you are using elf or linux gdb.

If you are trying to use a riscv-linux native gdb, this does not work in riscv-gnu-toolchain, but it does work in upstream FSF gdb development sources, provided that you have the necessary linux kernel patches. We may try to switch to FSF gdb when the gdb-8.2 release occurs. See for instance https://github.com/jim-wilson/riscv-linux-native-gdb/blob/jimw-riscv-linux-gdb/README.md This has been tested on both a Hifive Unleashed board, and system mode qemu, both running a linux kernel with the gdb patches applied.

If you are trying to use an elf gdb, then that should work on hardware with OpenOCD, but I don't know if that works with qemu. It should also work with spike.

[lbmeng]

Please check below the reproduce steps:

• Follow README.md in https://github.com/riscv/riscv-gnu-toolchain, to build Linux multilib toolchain for riscv-v. This will build riscv64-unknown-linux-gnu-gcc and riscv64-unknown-linux-gnu-gdb.
• cd riscv-qemu; ./configure --target-list=riscv64-softmmu; to build riscv-qemu
• Follow README.md in https://github.com/riscv/riscv-pk to build bbl (with -g option)
• Execute: qemu-system-riscv64 -nographic -machine virt -kernel bbl -s -S
• Execute: riscv64-unknown-linux-gnu-gdb bbl (from another shell)

  target remote :1234 b hart_init Register 3921 is not available info b No breakpoints or watchpoints.

[jim-wilson]

If you are using system qemu, you can run gdb directly from the shell. That is known to work, provided that you have FSF gdb top of tree (don't use riscv-gnu-toolchain gdb) and the necessary linux kernel patches (3 of 4 in 4.19). This is how I use gdb. This won't work for kernel debugging of course.

The error means that gdb is trying to read misa, failing, catching the exception, trying to read the old v1.9 misa, failing, and that exception is not caught causing the command to fail. The ISA spec says that misa must always exist and always be readable, but implementations are allowed to return zero for a misa read. So failing for a misa read is a broken implementation. My FSF gdb riscv-linux patches just return 0 for a misa read, as I don't have ptrace support for it yet. Gdb then uses ELF headers to determine system features instead of misa. This suggests a problem on the qemu side with reading system registers. There is another gdb issue that talks about this. See for instance #156. This is likely the same problem.

Possible workarounds are to hack gdb to pretend that misa is always zero, or hack qemu to special case reads of misa, as that is the only system register gdb needs. Long term, we do needs reads of misa to work for proper functioning of gdb.

[michaeljclark]

On Tue, 4 Sep 2018 at 7:06 AM, Jim Wilson notifications@github.com wrote:

If you are using system qemu, you can run gdb directly from the shell. That is known to work, provided that you have FSF gdb top of tree (don't use riscv-gnu-toolchain gdb) and the necessary linux kernel patches (3 of 4 in 4.19). This is how I use gdb. This won't work for kernel debugging of course.

The error means that gdb is trying to read misa, failing, catching the exception, trying to read the old v1.9 misa, failing, and that exception is not caught causing the command to fail. The ISA spec says that misa must always exist and always be readable, but implementations are allowed to return zero for a misa read. So failing for a misa read is a broken implementation. My FSF gdb riscv-linux patches just return 0 for a misa read, as I don't have ptrace support for it yet. Gdb then uses ELF headers to determine system features instead of misa. This suggests a problem on the qemu side with reading system registers. There is another gdb issue that talks about this. See for instance #156 https://github.com/riscv/riscv-qemu/issues/156. This is likely the same problem.

Possible workarounds are to hack gdb to pretend that misa is always zero, or hack qemu to special case reads of misa, as that is the only system register gdb needs. Long term, we do needs reads of misa to work for proper functioning of gdb.

It should be relatively easy to fix CSR read support. There has been some work done on debug support for system mode in the riscv-qemu repo: https://github.com/riscv/riscv-qemu

I’m curious if folk could test against the riscv-qemu tree. One of the issues is that the CSR mappings in earlier versions of GDB may not be 100% stable. I remember Jim mentioning that old GDB versions had CSR numbers from earlier versions of the Privileged ISA spec.

I’m also keen to get on the new GDB. I might spend some time looking at this if I can find the right GDB tree with all the goodness like Jim’s native Linux debug support. Hopefully https://github.com/riscv-gdb ?

The riscv-qemu tree has a new modular CSR system with support for atomic CSRs and it decouples the CSR implementation from the interface (previously there were two big switch statements like spike aka riscv_isa_sim), and we have built on top of this new support, I think it is still both unreviewed and unchanged since it was last sent to qemu-devel.

The rationale, beyond atomic CSRs and CSRs that carry a dependency from rs1 to rd, which is now possible (which on their own, required a new interface) was so that we could eventually modularise handling of CSRs to core specific code. In the case of SiFive, we want to change mcause/scause handling and add some new CSRs for a new interrupt controller. We can now have different core features hook the CSR table.

In fact, the result is more generic code that could implement other RISC-V cores that do not implement the Privileged ISA specification but have their own non-standard CSRs. I know have to think about cores that have a different CPU state structures which is why we have an unmerged PR in the riscv-qemu tree.

[jim-wilson]

The gdb CSR mappings are stable. It is 65+CSR number. Maybe you are confused by the reference to old misa. We check current misa first, then old misa just in case we are talking to old hardware. But since qemu support is broken, both access attempts fail, and we end up reporting an error against old misa.

To look at current gdb sources, see sourceware.org/gdb. Eventually this will replace riscv-[binutils-]gdb which still has the old gdb sources. There is some stuff that works better in the new FSF tree, and some stuff that works better in the old tree. When this is fixed, the old tree will be replaced with the new one. My riscv-linux native stuff is only in the FSF tree. The FSF gdb tree is the only one interesting to me.

[jim-wilson]

I see that there are some fixes to qemu riscv_cpu_gdb_read_register() on the riscv-all branch, so maybe that will work with gdb. But issue #156 is still open with no comments, so I don't know what is going on. Just that gdb isn't broken here.

[michaeljclark]

On Tue, 4 Sep 2018 at 11:35 AM, Jim Wilson notifications@github.com wrote:

I see that there are some fixes to qemu riscv_cpu_gdb_read_register() on the riscv-all branch, so maybe that will work with gdb. But issue #156 https://github.com/riscv/riscv-qemu/issues/156 is still open with no comments, so I don't know what is going on. Just that gdb isn't broken here.

I’ll do some testing with GDB against QEMU in full system mode...

[jim-wilson]

I'm trying to reproduce too.

cd riscv-qemu; ./configure --target-list=riscv64-softmmu; to build riscv-qemu

@lbmeng Are you using the riscv-qemu from riscv-gnu-toolchain? That is a mistake. glibc is only half upstream (64-bit but not 32-bit), and there are ABI issues (e.g. stat structure) with the 32-bit support that need to be resolved, as a result riscv-gnu-toolchain is deliberately using an old obsolete qemu and an old obsolete glibc. We can't upgrade to upstream glibc as then we lose 32-bit linux support. And we can't upgrade to current qemu as that breaks 32-bit linux support due to the stat structure issue. This will take a few more months to fix. Meanwhile, the riscv-qemu in riscv-gnu-toolchain should not be used for anything other than running the binutils and gcc testsuites. You need to use the riscv-qemu master branch instead. The only good parts of riscv-gnu-toolchain are binutils, gcc, and newlib. Gdb and glibc may or may not be good depending on what you are doing as those are both only partly upstream. qemu is not considered part of the toolchain, it is only included for testing purposes.

[jim-wilson]

@lbmeng Maybe I took your reproduce instructions a little too literally. You didn't mention cloning riscv-qemu, so I tried using the one I already had in riscv-gnu-toolchain. But that doesn't support -machine virt so that can't be the one you are using. I tried cloning riscv/riscv-qemu instead. Building riscv-gnu-toolchain --enable-linux --enable-multilib doesn't work anymore because of an ABI conflict between current binutils and the old obsolete glibc we have. I configured for --with-abi=lp64 instead for the riscv-pk build.

I reproduced the problem. The first thing I noticed is that while target/riscv/gdbstub.c appears to have support for CSRs, gdbstub.c has if (reg < cc->gdb_num_core_regs) { return cc->gdb_read_register(cpu, mem_buf, reg); } and target/riscv/cpu.c has cc->gdb_num_core_regs = 65; so the target gdbstub is never called for CSRs. If I change cc->gdb_num_core_regs to 65 + 4096, then I see that gdb tries to read every single one when it connects to the target, and then qemu core dumps. Unfortunately, I can't get a useful backtrace from qemu even though I added -g to CFLAGS, CXXCFLAGS, etc, in config-host.mk. I will have to debug the qemu build to see what is going on, but it is getting late here, so I will do that some other time.

I also haven't tried FSF gdb yet, which may or may not be better than riscv-gnu-toolchain gdb.

[michaeljclark]

On Tue, Sep 4, 2018 at 3:05 PM, Jim Wilson notifications@github.com wrote:

I'm trying to reproduce too.

cd riscv-qemu; ./configure --target-list=riscv64-softmmu; to build riscv-qemu

I am able to debug programs in full system mode with GNU gdb (GDB) 8.0.50.20170724-git and riscv-qemu v3.0.0-rc2-44-g5da132f38ddc-dirty (the riscv-qemu repo version with all of our bug fix patches)

I added a simple bare-metal hang program to riscv-probe here: https://github.com/michaeljclark/riscv-probe (note: riscv-probe can be used to check CSR existance, which variers between machine). I was able to debug it. No errors about missing CSRs, however I did have trouble detaching.

$ qemu-system-riscv32 -s -nographic -machine sifive_e -kernel build/bin/rv32/qemu-sifive_e/hang hang

$ riscv64-unknown-elf-gdb build/bin/rv32/qemu-sifive_e/hang GNU gdb (GDB) 8.0.50.20170724-git Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html

This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-pc-linux-gnu --target=riscv64-unknown-elf". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from build/bin/rv32/qemu-sifive_e/hang...done. (gdb) target extended-remote localhost:1234 Remote debugging using localhost:1234 main () at examples/hang/hang.c:10 10 a = b; (gdb) si 22 m = a; (gdb) si 0x20400104 22 m = a; (gdb) si 0x20400106 22 m = a; (gdb) si 0x20400108 22 m = a; (gdb) si 0x2040010a 22 m = a; (gdb) c Continuing. ^C Program received signal SIGINT, Interrupt. main () at examples/hang/hang.c:10 10 a = b; (gdb) info registers ra 0x20400100 541065472 sp 0x800003d0 2147484624 gp 0x00000000 0 tp 0x00000000 0 t0 0x00000000 0 t1 0x00000000 0 t2 0x00000000 0 s0 0x00000000 0 s1 0x00000000 0 a0 0x0000000a 10 a1 0x00000000 0 a2 0x00000000 0 a3 0x00000000 0 a4 0x00000000 0 a5 0x00000000 0 a6 0x00000000 0 a7 0x00000000 0 s2 0x00000000 0 s3 0x00000000 0 s4 0x00000000 0 s5 0x00000000 0 s6 0x00000000 0 s7 0x00000000 0 s8 0x00000000 0 s9 0x00000000 0 s10 0x00000000 0 s11 0x00000000 0 t3 0x00000000 0 t4 0x00000000 0 t5 0x00000000 0 t6 0x00000000 0 pc 0x20400100 541065472 priv [Invalid] (gdb) quit A debugging session is active.

Inferior 1 [Remote target] will be detached.

Quit anyway? (y or n) y Detaching from program: /home/mjc/src/sifive/riscv-probe/build/bin/rv32/qemu-sifive_e/hang, Remote target ^C^CThe target is not responding to GDB commands. Stop debugging it? (y or n) y Disconnected from target.

@lbmeng https://github.com/lbmeng Are you using the riscv-qemu from

riscv-gnu-toolchain? That is a mistake. glibc is only half upstream (64-bit but not 32-bit), and there are ABI issues (e.g. stat structure) with the 32-bit support that need to be resolved, as a result riscv-gnu-toolchain is deliberately using an old obsolete qemu and an old obsolete glibc. We can't upgrade to upstream glibc as then we lose 32-bit linux support. And we can't upgrade to current qemu as that breaks 32-bit linux support due to the stat structure issue. This will take a few more months to fix. Meanwhile, the riscv-qemu in riscv-gnu-toolchain should not be used for anything other than running the binutils and gcc testsuites. You need to use the riscv-qemu master branch instead. The only good parts of riscv-gnu-toolchain are binutils, gcc, and newlib. Gdb and glibc may or may not be good depending on what you are doing as those are both only partly upstream. qemu is not considered part of the toolchain, it is only included for testing purposes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/riscv/riscv-qemu/issues/101#issuecomment-418227714, or mute the thread https://github.com/notifications/unsubscribe-auth/AAV25y3BRf0qa3p6iTpKgYdFd1yuDZEIks5uXe3ugaJpZM4R8uau .

[lbmeng]

@jim-wilson

Are you using the riscv-qemu from riscv-gnu-toolchain?

Yes, I was building riscv-qemu from riscv-all branch.

Meanwhile, the riscv-qemu in riscv-gnu-toolchain should not be used for anything other than running the binutils and gcc testsuites.

Too bad. If that's the case, can we update the riscv-gnu-toolchain git repo so that people know?

I was currently using riscv64-unkown-linux-gnu-gcc to build both BBL and Linux kernel and it works. I did not test glibc though.

[jim-wilson]

@michaeljclark Your gdb log doesn't show you trying to set a breakpoint, or trying to read a CSR, e.g. "print $misa". Those are the commands that will fail.

[jim-wilson]

@lbmeng Update riscv-gnu-toolchain how? It doesn't build and install qemu unless you use make check, and only developers are likely to use that. Packages that need riscv-qemu, like freedom-u-sdk, have their own copy of qemu, they don't use the one from riscv-gnu-toolchain.

[lbmeng]

@jim-wilson

Sorry, I mean update riscv-gnu-toolchain's README.md to mention that only this and that are supported. Or just remove riscv-qemu/riscv-gdb from the git repo, and point people where to download the working one.

[jim-wilson]

riscv-gdb does work, but some things work better in the FSF tree, and some things work better in riscv-gdb. Gdb is in a partially upstreamed state. Upstreaming is important, and when upstreaming is finished there will be just the one gdb port. Meanwhile, we are stuck dealing with a bad situation that we are working around as best as possible.

riscv-qemu does work, and is in fact the preferred source tree because of problems with upstream being slow to accept patches from us. riscv-gnu-toolchain just happens to use an old obsolete version because glibc is in a partially upstreamed state. Some things work better in the FSF tree and some things work better in the riscv-glibc tree. Upstreaming is important, and when upstreaming is finished there will be just the one glibc port. Meanwhile, we are stuck dealing with a bad situation that we are working around as best as possible.

Basically, from my point of view, there is no easy way to explain what is going on, and the current state is a temporary state from being in the middle of upstreaming, which is important enough that we should not do anything that interferes with the upstreaming, and need to work around any inconveniences as best as possible. You can avoid some of the confusion by just using FSF source trees and ignoring the riscv- versions of FSF tools. You probably don't need any of the stuff that is still in the riscv- toolchain trees. I do find riscv-gnu-toolchain convenient as a build script though.

[lbmeng]

So where can I find a working riscv-gdb that can set breakpoint without the error message "Register 3921 is not available" with qemu-riscv system mode?

[jim-wilson]

It is a qemu problem, not a gdb problem. The qemu riscv gdb stuabs don't support reading CSRs. So there is no gdb that is going to work. Though you could hack gdb to pretend that misa is always 0, which would be OK if you never need to read any CSRs. The real solution is the one mentioned in issue #156: we need to add XML register support to qemu. I took a stab at this, and it ended up being a bigger patch that I expected, and I don't have it working yet. I'm at the GNU Cauldron/vacation, so don't expect any progress for a week.

[jim-wilson]

See PR #160 for a possible solution. This patch needs more work before it can be accepted.

[lbmeng]

Thanks Jim. I can confirm PR#160 solves my problem, at least I can set the breakpoint when using remote debug with gdb/qemu.

[lbmeng]

See PR #160 for a possible solution. This patch needs more work before it can be accepted.

Jim, any plan to address the s-mode registers display in QEMU?

[jim-wilson]

I've been busy with FSF gdb, but a FSF gdb xml register set patch was just posted, which means I will want to get qemu working with FSF gdb now, which will require extending the xml register set support in qemu. This is on my short list now, but I've got a lot of stuff on my to do list so not sure when I will get to it. I will also want to replace the riscv-gnu-toolchain gdb with an FSF gdb sometime soon.

[fintelia]

What is the status of this? Is there any combination of qemu and gdb for RISC-V that supports setting breakpoints?

[zakkak]

It looks like FSF gdb 8.2.1 with riscv-qemu (git tag riscv-for-master-3.1-rc2) doesn't have this issue. You can get gdb from https://www.gnu.org/software/gdb/download/ and build with:

./configure --target=riscv32-elf,riscv64-elf

Edit: I forgot to mention the qemu version

[jim-wilson]

This qemu is no longer actively maintained. You should consider using the official upstream one at qemu.org.

I posted two patches for this riscv-qemu repo before I started working upstream. The first patch is pull request #160 which is mentioned multiple times above, and I think was merged in somewhere. The second patch is pull request #192 which may never be merged in as this repo is not actively maintained anymore.

Meanwhile, upstream, I have a 5-patch set that has already been approved and is waiting for other in progress patches to be finished before it finally gets merged in. http://lists.nongnu.org/archive/html/qemu-devel/2019-02/msg03143.html

All of these require a gdb that has RISC-V XML support, which was added to the FSF GDB tree in November of last year. This is the gdb in current riscv-gnu-toolchain as of about November last year, so if you have an up to date toolchain and a patched qemu it should work.

[jim-wilson]

Sorry, the second patch for this repo is #182.

[zakkak]

@jim-wilson I think we can close this issue then

[jim-wilson]

I was waiting for the upstream patches to get merged in before abandoning these patches.

[arunthomas]

@jim-wilson Do you expect these patches will be merged in time for the 4.0 release?

[jim-wilson]

I'm not a qemu developer, and don't know anything about qemu releases. I believe the gdbstub patches are next in line after the d-tree decoder patches, and may already be on a merge/integration branch somewhere.

[zakkak]

I just tested with qemu 3.1.0 from (https://www.qemu.org/download/) and gdb 8.2.1 from (https://www.gnu.org/software/gdb/download/) and everything works fine for me

[jim-wilson]

There is some RISC-V debug support in qemu 3.1.0, and some RISC-V debug support in gdb 8.2, but neither is correct and complete. They may work for some targets, where qemu and gdb agree on the register sets, and they may work for some use cases, if you only try to view registers that qemu and gdb use the same register number for. But this support will not work for all targets and all use cases. Try using a system qemu and typing "info registers misa" for instance.

There are patches that will arrive in gdb 8.3 and (hopefully) qemu 4 that will greatly improve this situation, and allow this to work for many more targets and use cases.

[jim-wilson]

The gdbstub patches are now in upstream qemu, and I expect that they will appear in the qemu 4.0 release. This issue can be closed now, though I don't have write access here so I am not able to do that.

[zakkak]

Thank you @jim-wilson