probe-run timeouts when trying to run an example from knurling-rs template

[laplab]

Describe the bug

At attempt to run hello example from knurling-rs template results in an "Operation timed out" error.

To Reproduce

1. Set up two Raspberry Pi Pico W according to the wiring instructions
2. Flash the probe Pico with DapperMime firware
3. Generate example project from the app-template using cargo-generate
4. Fix all the todos. Use rp2040-hal = "0.6.0" HAL and RP2040 for the --chip argument of probe-run. Also copy the memory.x file into the project root.
5. Try running cargo run --release --bin hello

Expected and observed behavior

Expected: program flashes the uf2, opens defmt session, displays "Hello, world!" and exits.

Observed:

$ cargo run --release --bin hello
    Finished release [optimized + debuginfo] target(s) in 0.18s
     Running `probe-run --chip RP2040 target/thumbv6m-none-eabi/release/hello`
(HOST) INFO  flashing program (2 pages / 8.00 KiB)
(HOST) INFO  success!
Error: An error with the usage of the probe occurred

Caused by:
    Operation timed out

config.toml

[target.'cfg(all(target_arch = "arm", target_os = "none"))']
runner = "probe-run --chip RP2040"
rustflags = [
  "-C", "linker=flip-link",
  "-C", "link-arg=-Tlink.x",
  "-C", "link-arg=-Tdefmt.x",
  # This is needed if your flash or ram addresses are not aligned to 0x10000 in memory.x
  # See https://github.com/rust-embedded/cortex-m-quickstart/pull/95
  "-C", "link-arg=--nmagic",
]

[build]
target = "thumbv6m-none-eabi"    # Cortex-M0 and Cortex-M0+

[alias]
rb = "run --bin"
rrb = "run --release --bin"

Probe details

$ probe-run --list-probes
the following probes were found:
[0]: RP2040 CMSIS-DAP (VID: cafe, PID: 4005, Serial: 6E16144030E45C33, CmsisDap)

Operating System:

macOS Monterey 12.4

ELF file (attachment)

hello.zip

Additional context

I have tried to get additional context on the issue before reporting it. I cloned probe-run and probe-rs repositories and accompanied some occurrences of DebugProbeError::Timeout error with panic! to get the origin of the error. Here is what I got when running modified version of probe-run with -v flag passed:

RUST_BACKTRACE=1 cargo run --release --bin hello
    Finished release [optimized + debuginfo] target(s) in 0.08s
     Running `/Users/laplab/Desktop/dbg-test/probe-run/target/x86_64-apple-darwin/debug/probe-run -v --chip RP2040 target/thumbv6m-none-eabi/release/hello`
(HOST) DEBUG vector table: VectorTable { initial_stack_pointer: 2003fbc8, hard_fault: 10001617 }
└─ probe_run::elf @ src/elf.rs:31
(HOST) DEBUG RAM region: 0x20000000-0x2003FFFF
└─ probe_run::target_info @ src/target_info.rs:115
(HOST) DEBUG section `.data` is in RAM at 0x2003FBC8..=0x2003FBF7
└─ probe_run::target_info @ src/target_info.rs:153
(HOST) DEBUG section `.bss` is in RAM at 0x2003FBF8..=0x2003FBFF
└─ probe_run::target_info @ src/target_info.rs:153
(HOST) DEBUG section `.uninit` is in RAM at 0x2003FC00..=0x2003FFFF
└─ probe_run::target_info @ src/target_info.rs:153
(HOST) DEBUG valid SP range: 0x20000000..=0x2003FBC4
└─ probe_run::target_info @ src/target_info.rs:167
(HOST) DEBUG found 1 probes
└─ probe_run::probe @ src/probe.rs:25
(HOST) DEBUG opened probe
└─ probe_run::probe @ src/probe.rs:33
(HOST) DEBUG started session
└─ probe_run @ src/main.rs:86
(HOST) INFO  flashing program (2 pages / 8.00 KiB)
└─ probe_run @ src/main.rs:98
(HOST) DEBUG Erased sector of size 4096 bytes in 134 ms
└─ probe_run @ src/main.rs:106
(HOST) DEBUG Erased sector of size 4096 bytes in 76 ms
└─ probe_run @ src/main.rs:106
(HOST) DEBUG Programmed page of size 4096 bytes in 227 ms
└─ probe_run @ src/main.rs:114
(HOST) DEBUG Programmed page of size 4096 bytes in 223 ms
└─ probe_run @ src/main.rs:114
(HOST) INFO  success!
└─ probe_run @ src/main.rs:132
(HOST) DEBUG 261060 bytes of stack available (0x20000000 ..= 0x2003FBC4), using 1024 byte canary
└─ probe_run::canary @ src/canary.rs:87
(HOST) TRACE setting up canary took 0.040s (25.02 KiB/s)
└─ probe_run::canary @ src/canary.rs:101
(HOST) DEBUG starting device
└─ probe_run @ src/main.rs:190
thread 'main' panicked at 'laplab', probe-rs/probe-rs/src/architecture/arm/core/armv6m.rs:514:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/c07a8b4e09f356c7468b69c50cac7fc5b5000b8a/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/c07a8b4e09f356c7468b69c50cac7fc5b5000b8a/library/core/src/panicking.rs:142:14
   2: <probe_rs::architecture::arm::core::armv6m::Armv6m as probe_rs::core::CoreInterface>::wait_for_core_halted
             at ./probe-run/probe-rs/probe-rs/src/architecture/arm/core/armv6m.rs:514:9
   3: probe_rs::core::Core::wait_for_core_halted
             at ./probe-run/probe-rs/probe-rs/src/core/mod.rs:767:9
   4: probe_run::set_rtt_to_blocking
             at ./probe-run/src/main.rs:218:5
   5: probe_run::start_program
             at ./probe-run/src/main.rs:200:9
   6: probe_run::run_target_program
             at ./probe-run/src/main.rs:139:5
   7: probe_run::cli::handle_arguments
             at ./probe-run/src/cli.rs:120:9
   8: probe_run::main
             at ./probe-run/src/main.rs:43:5
   9: core::ops::function::FnOnce::call_once
             at /rustc/c07a8b4e09f356c7468b69c50cac7fc5b5000b8a/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

After reading the probe-run code, I figured out that the hardware breakpoint set for the main function was not hit for some reason, since the timeout is coming from this line. At this point, I decided to create a Github Issue.

[laplab]

In at attempt to get more information, I added a debug print of this dhcsr value read by probe-rs.

After this wait_for_core_halted is called, the history of dhcsr values is the following (I explicitly mention "true" values, all others are false):

1. s_retire_st and c_debugen are set for a couple of iterations
2. On one of the iterations, three flags s_retire_st, s_sleep and c_debugen are set
3. After that, only s_sleep and c_debugen are set

Hope this helps.

[Urhengulas]

Thank you for reporting. I don't have a pi pico w at hand to test it. Does it also occur with the pi pico (not W)?

[laplab]

I am afraid I only have Pico W at hand. If it helps, I am open to live debugging session over Zoom 🙂

[Urhengulas]

I am afraid I only have Pico W at hand.

I might be able to get my hands on one. I will try.

If it helps, I am open to live debugging session over Zoom slightly_smiling_face

Let's keep this as an option!

[VanillaBrooks]

Does it also occur with the pi pico (not W)?

I tested on two regular pico's (pico A debugging B, and B debugging A) and could not reproduce the issue. However, I created a custom board using RP 2040 and W25Q128JVSIQ as flash (pico uses W25Q16JVUXIQ) that does experience the exact same timeout in the exact same place (as far as I can tell).

[Urhengulas]

I tested on two regular pico's (pico A debugging B, and B debugging A) and could not reproduce the issue. However, I created a custom board using RP 2040 and W25Q128JVSIQ as flash (pico uses W25Q16JVUXIQ) that does experience the exact same timeout in the exact same place (as far as I can tell).

Thank you for your investigation. As far as I can read from the data sheets pico and pico 2 use the same flash, so I am not sure if the problem stems from there... 🤔

[Laura7089]

I'm also having this issue - using an RPi Pico H with DapperMime as a probe, debugging a custom RP2040-based board.

[jonathanpallant]

Are you using an RP2040 BSP, or do you have the RP2040 bootloader set up? The RP2040 doesn't boot like any other Cortex-M CPU and requires a second-stage bootloader. Please see https://github.com/rp-rs/rp2040-boot2 for more details. The HAL cannot include the bootloader because it varies depending on the Flash chip you have, which is why it is included in each of the various BSPs; or you can include it directly in your main.rs.

[jonathanpallant]

@Urhengulas we should add a note to app-template that it doesn't work on the RP2040 and that they need to use an RP2040-based BSP, or otherwise include the Bootloader.

[laplab]

@jonathanpallant Adding the bootloader from https://github.com/rp-rs/rp2040-boot2 fixed the issue for me. Thank you so much!

[Urhengulas]

Thank you Jonathan!

@laplab said:

@jonathanpallant Adding the bootloader from https://github.com/rp-rs/rp2040-boot2 fixed the issue for me. Thank you so much!

Great!

@jonathanpallant said:

@Urhengulas we should add a note to app-template that it doesn't work on the RP2040 and that they need to use an RP2040-based BSP, or otherwise include the Bootloader.

Yes. We can also refer to https://github.com/rp-rs/rp2040-project-template/ for that.

[VanillaBrooks]

@jonathanpallant said:

Are you using an RP2040 BSP, or do you have the RP2040 bootloader set up?

I initially used a BSP to verify that two rp pico H boards could debug each other, then I manually setup the rp2040-boot2 bootloader with rp2040_boot2::BOOT_LOADER_W25Q080 to verify I was working was configuring the bootloader correctly without the help of the board crate.

Moving to my custom rp2040 based board I tried both rp2040_boot2::BOOT_LOADER_W25Q080 and rp2040_boot2::BOOT_LOADER_GENERIC_03H (my flash is W25Q128JVSIQ which was recommended in the "Hardware design with RP2040" pdf), which did not work

I was mainly working from the rp2040-project-template repo linked above, but since I could not reproduce the issue on an rp pico H w/ either bootloader I assumed it must be a probe-run issue.

@Laura7089 what flash are you using? And did did the bootloader suggestion fix your issue?