search

raspberrypi / pico-sdk

BSD 3-Clause "New" or "Revised" License
3.26k stars 838 forks source link

Improve documentation for (and clarify) timers freezing execution during debugging #1622

Open recursivenomad opened 5 months ago

recursivenomad commented 5 months ago

In my experience debugging with stock OpenOCD through VSCode on Windows, timers do not increment; and as a result all sleep functions cause the debugged program to freeze. See for more info:

Two separate solutions/workarounds offered in those threads are:

In light of this, I am seeking clarification on the following points:

  1. Is it intended for cores to pause their timers the entire time they're being debugged?
  2. Is it intended for all sleep functions to be incompatible with debugging by default?
  3. Does this problem occur in Raspberry Pi's fork of OpenOCD?
  4. What does set USE_CORE 0 explicitly do, and is it safe/scalable?
  5. What is the recommended solution?

It would also be helpful if these points could make their way into official documentation, as to prevent confusion around this in the future.

peterharperuk commented 5 months ago

I have no problem with timers when I'm debugging on Linux. It suggests I'm using 0.12.0. I don't use USE_CORE or anything else. @P33M does this mean anything to you?

lurch commented 5 months ago

From what little I understand of the issue, I believe this is all related to the DBGPAUSE register, documented on page 544 of https://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdf

recursivenomad commented 5 months ago

It suggests I'm using 0.12.0.

I am also using 0.12.0 of stock OpenOCD (on Windows)

From what little I understand of the issue, I believe this is all related to the DBGPAUSE register, documented on page 544 of https://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdf

That's my best guess too, that register is what setting timer_hw->dbgpause = 0; at the top of my code is disabling.

There's a chance that running OpenOCD and gdb manually (as opposed through VSCode) sometimes allows me to continue over timers? But I have not been able to produce consistent results with that yet; I may continue trying. In the meantime, as this issue is experienced by more than just myself, I'd still like to try and get to the bottom of what's going on here and why setting DBGPAUSE to 0 or passing -c set USE_CORE 0 to OpenOCD "fixes" it.

A minimum example to produce the issue through VSCode's debugger on my machine: ```c // main.c #include #include "pico/stdlib.h" int main() { //timer_hw->dbgpause = 0; stdio_init_all(); while (true) { printf("DBGPAUSE register value: %d\n", timer_hw->dbgpause); sleep_ms(1000); } } ``` ```cmake cmake_minimum_required(VERSION 3.13) set(PICO_DEOPTIMIZED_DEBUG ON) include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake) project(main) pico_sdk_init() add_executable(${PROJECT_NAME} main.c) target_link_libraries(${PROJECT_NAME} pico_stdlib) pico_enable_stdio_uart(${PROJECT_NAME} ON) pico_add_extra_outputs(${PROJECT_NAME}) ``` ```json "launch": { "version": "0.2.0", "configurations": [ { "name": "Pico Debug", "cwd": "${workspaceRoot}", "executable": "${command:cmake.launchTargetPath}", "request": "launch", "type": "cortex-debug", "servertype": "openocd", "gdbPath" : "arm-none-eabi-gdb", "device": "RP2040", "configFiles": [ "interface/cmsis-dap.cfg", "target/rp2040.cfg" ], "openOCDLaunchCommands": [ "adapter speed 5000" ], "svdFile": "${env:PICO_SDK_PATH}/src/rp2040/hardware_regs/rp2040.svd", "runToEntryPoint": "main", "postRestartCommands": [ "break main", "continue" ], } ] } ```

With the above example, debugging will initially wait for user input at stdio_init_all();. Then pressing continue, my serial monitor receives DBGPAUSE register value: 7 and the execution freezes. If you then pause debugging, it will show that it's locked in the function sleep_until() at line 398 of pico-sdk/src/common/pico_time/time.c:

            while (!time_reached(t_before)) {
                uint32_t save = spin_lock_blocking(sleep_notifier.spin_lock);
                lock_internal_spin_unlock_with_wait(&sleep_notifier, save);
            }

Uncommenting timer_hw->dbgpause = 0; in main.c resolves the issue for me, and repeats DBGPAUSE register value: 0 as expected.

EDIT: My build environment is: - GCC 13.2.0 (MinGW-w64) - CMake 3.28.1 - `-G "MinGW Makefiles"` - Python 3.12.1 - arm-none-eabi-gcc 13.2.1 - arm-none-eabi-gdb 13.2.90.20231008-git - Pico SDK 1.5.1 - OpenOCD 0.12.0 - Picoprobe built from commit hash `721b69cf5c8535e57995dbdd2e74f1bbc2f36944` - VSCodium 1.84.2.23317 - CMake Tools extension 1.16.32 - Cortex-Debug extension 1.12.1
recursivenomad commented 5 months ago

I have identified a third workaround - passing monitor reset to gdb (the "DEBUG CONSOLE" tab in VSCode) when the timers are erroniously paused allows me to debug as inteded with sleep_ms() functioning as expected (until the next reset), no supplemental OpenOCD commands nor modified timer registers needed. It's not perfect, as monitor reset appears to skip over certain breakpoints when it is called, but perhaps it can provide another clue to what's going on here.

It's worth noting that passing "monitor halt" is fine, but passing "monitor reset halt" (which is what Cortex-Debug/gdb appears to be sending already) places the timers in a frozen state again. This is the exact same behaviour as documented in this comment here:

So to recap, to my understanding whatever is causing this issue appears to involve:

Perhaps this issue belongs in raspberrypi/picoprobe rather than the sdk, I'm starting to think it's more related to how the physical debugger (or target RP2040?) is handling halting... Unfortunatly I do not have another CMSIS-DAP device to compare with.

peterharperuk commented 5 months ago

Hmm, you might be on to something. I always run "monitor reset init" after loading the elf. I forgot about that because I have it in .gdbinit and I have some tools to generate this for me automatically.

recursivenomad commented 5 months ago

So I gave monitor reset init a try, but unfortunately that does not resolve the frozen timers in my environment either. So far for gdb commands, only the monitor reset command (sorta) resolves the issue for me. So something's going on where the halting baked-in to monitor reset halt and monitor reset init is inappropriately halting the timers, but where monitor halt does not cause any apparent issues.

(This exclusivity of only monitor reset working while monitor reset halt and monitor reset init cause this specific problem is the same for the commenter of picoprobe/#45)

@peterharperuk I know you're on Linux and I'm on Windows, which is a major variable; but that aside, could you confirm that the 0.12.0 OpenOCD you're running is stock OpenOCD (such as from a package manager), and that you didn't build it from source from the Raspberry Pi fork?

recursivenomad commented 5 months ago

Update: I spun up a fresh install of Debian Bookworm this weekend, and I can confirm that this timer freezing occurs in Debian as well, even when following the exact instructions in Getting started with the Raspberry Pi Pico (the only change being swapping -f interface/raspberrypi-swd.cfg with -f interface/cmsis-dap.cfg). Uploading & running "blink" or "hello_world/serial" results in continue advancing to the first timer, and then freezing as described above.

So it's not limited to Windows, and even occurs in the suggested Debian environment. I also didn't get to installing VSCode in this test yet, so I believe the problem is isolated to the interaction between GDB, OpenOCD, Picoprobe, and the RP2040 itself.

If anyone has the time to reproduce this behaviour in a fresh Debian environment, knowing it's not just me would give me peace of mind to know I'm pursuing a real issue here. In the meantime, I think I'm going to do a binary search through Picoprobe's and the Pico SDK's commit histories to see if I can find a point which works as expected...

Build environment: - Debian 12 - GCC 12.2.0 - CMake 3.25.1 - arm-none-eabi-gcc 12.2.1 - gdb-multiarch (Debian 13.1-3) 13.1 - Pico SDK 1.5.1 - OpenOCD 0.12.0 *(from apt, not built from source)* - Picoprobe built from commit hash `721b69cf5c8535e57995dbdd2e74f1bbc2f36944`

EDITS:

I am finding that regardless of the state the Pico is in (halted, continuing as intended, frozen in a timer, etc), submitting monitor reset halt or monitor reset init to GDB will always get it to lock up when it reaches a timer, without fail.

@peterharperuk, could I ask you to run a program with timers (such as pico-examples/blink) through a Picoprobe, and when it's debugging as expected to enter monitor reset halt continue into GDB? If your timers don't lock up from this, I'd be very curious to know the specifics of your build environment and GDB/OpenOCD configurations...

recursivenomad commented 4 months ago

I understand the cause of the issue!

Stock OpenOCD 0.12.0, as available through apt or the pre-built Windows binary, is the problem.

Specifically commits e877d52 and 9a9dbf8 in Raspberry Pi's OpenOCD fork, implementing SMP support, are the two commits that fixed this issue. These commits have both been merged into the main OpenOCD repository. In fact, the only commits that have not been merged from the fork are the 3 most recent commits, seemingly pertaining to debugging from a Raspberry Pi SBC directly; everything else has been merged into the main OpenOCD repository.

To resolve the problem, I built OpenOCD from source on Debian, and grabbed the latest snapshot build from the official OpenOCD repository (specifically the artifact of commit 4593c75) for Windows - everything works as intended with those 🎉

I'm inclined not to close this issue yet, as the documentation still recommends installing OpenOCD through apt, and I believe it would be useful to mention that this issue will occur if your OpenOCD version does not contain RP2040 SMP support.

peterharperuk commented 4 months ago

Thanks and well done. I'm still confused why I don't see this problem

recursivenomad commented 4 months ago

Do you develop on a Raspberry Pi? I have a hunch that maybe Raspberry Pi OS packages the custom fork of OpenOCD.

You can loosely check if your OpenOCD has SMP support by opening /usr/share/openocd/scripts/target/rp2040.cfg and checking if line 34 has set _USE_CORE { 0 1 } or set _USE_CORE SMP (the latter suggesting a fully-featured build of OpenOCD).

If you're on amd64 Debian and using OpenOCD 0.12.0 without SMP support, then I'd certainly be curious as well to know what's different between our environments that you don't encounter this issue 🤔

peterharperuk commented 4 months ago

Do you develop on a Raspberry Pi

Yes. I'll check, and maybe try rebuilding to see if I can reproduce the issue

lurch commented 4 months ago

Stock OpenOCD 0.12.0, as available through apt or the pre-built Windows binary, is the problem.

@recursivenomad By "pre-built Windows binary" do you mean the OpenOCD included in https://github.com/raspberrypi/pico-setup-windows , or does that version work "correctly"?

recursivenomad commented 4 months ago

By "pre-build Windows binary" I'm talking about the official OpenOCD release binaries as built by the OpenOCD Organization (specifically 0.12.0, which is the most recent version at the time of writing) at https://github.com/openocd-org/openocd/releases/tag/v0.12.0

But as noted above, the official OpenOCD automated snapshot builds (specifically the artifact from https://github.com/openocd-org/openocd/actions/runs/7771215993) work as intended, as they have had SMP support merged in.

I have not tried using Pico setup for Windows as I have prioritized my build environment software to be 100% open-source, and VSCode releases are not open-source, so I've opted for the manual setup to choose VSCodium (open-source builds of VSCode).

And yes, I understand the irony of wanting an open-source development environment whilst using Windows 😉

lurch commented 4 months ago

I have not tried using Pico setup for Windows as I have prioritized my build environment software to be 100% open-source, and VSCode releases are not open-source,

Pico-setup-windows doesn't include VSCode? https://github.com/raspberrypi/pico-setup-windows/issues/41

recursivenomad commented 4 months ago

Oh? I was going based off of the Getting started with Raspberry Pi Pico guide, which says VSCode is included so I avoided it.

lurch commented 4 months ago

Ahh, looking at https://github.com/raspberrypi/pico-setup-windows/blob/master/packages/pico-setup-windows/VSCodeUtils.nsh it seems that pico-setup-windows automatically downloads and installs the latest VSCode installer during setup. Apologies for the red herring! (and the off-topic discussion)

zarakshR commented 3 months ago

I can confirm that the Pi Pico freezes in this section of code -

while (!time_reached(t_before)) {
    uint32_t save = spin_lock_blocking(sleep_notifier.spin_lock);
    lock_internal_spin_unlock_with_wait(&sleep_notifier, save);
}

I am using the official Raspberry Pi Pico debug probe on Arch Linux using arm-none-eabi-gdb and openocd from the distro's official repository.

gdb only halts on the above section of code when it encounters a sleep_ms call.

CVasilakis commented 3 months ago

I also confirm that Pi Pico freezes in the section of code:

while (!time_reached(t_before)) {
    uint32_t save = spin_lock_blocking(sleep_notifier.spin_lock);
    lock_internal_spin_unlock_with_wait(&sleep_notifier, save);
}

I am also using the official Raspberry Pi Pico debug probe on WSL2 Debian (bookworm). I am also using arm-none-eabi-gdb and openocd installed using apt.

recursivenomad commented 3 months ago

@zarakshR @CVasilakis - My understanding is that OpenOCD 0.12.0 (the most recent release) does not support SMP, and this is causing the issue. I believe the only workaround on Linux is to build OpenOCD from source after commit 8af4d44, or waiting for 0.13.0 to release.