search

raspberrypi / openocd

Other
150 stars 90 forks source link

"OpenOCD init failed" - this has become a "hard failure" #116

Open seamusdemora opened 1 month ago

seamusdemora commented 1 month ago

I've had the Pico SDK installed on my RPi5 since about Jan or Feb this year (2024). To avoid the BOOTSEL procedure (which requires 3 hands :), I bought a "Raspberry Pi Debug Probe", and installed openocd. For the next several months, things worked very reliably. I use the RPi5 as my "development" machine; it also powers the Pico. The "Debug Probe" is also connected for uploading files to the Pico & driving the serial port.

The following openocd call became so routine that I placed a shortcut to it in my ~/.bashrc: 

sudo openocd -d0 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program ina260_trial.elf verify reset exit"

This worked extremely well for months. Recently, after having completed work on my 'INA260' project, I started on an "off-grid" project (Pico powered by small solar cell, battery). I decided to "upgrade" the SDK, and began using the sleep projects in pico-playground to reduce energy consumption for the "off-grid" project.

I had some difficulty with the setup of sleep (and dormant), but finally (seem to) have gotten that squared away. Then - openocd began failing with the error: OpenOCD init failed. It failed first with the hello_dormant example, and shortly thereafter it also failed on my original INA260 project - which was unchanged (except for the SDK upgrade)!!!

Searching online for answers didn't yield anything useful. I wondered if the SDK upgrade had something to do with it, and decided to update the firmware in the "Debug Probe" to the latest version - carefully following the instructions. That actually cured the error with the INA260 project, but not the hello_dormant project. After "thrashing about" for a while, making further changes to the hello_dormant project, openocd began failing again on the INA260 project. And I've been unable to recover after this last failure - re-flashing no longer resolves anything; i.e. this OpenOCD init failed error has become a "hard failure". Here's what it looks like with the -d2 option: 

$pwd 
~/pico/ina260_i2c/build

$ sudo openocd -d2 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program ina260_trial.elf verify reset exit"
Open On-Chip Debugger 0.12.0-g4d87f6d (2024-02-11-10:09)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
debug_level: 2

Info : Hardware thread awareness created
Info : Hardware thread awareness created
adapter speed: 5000 kHz

Info : Using CMSIS-DAPv2 interface with VID:PID=0x2e8a:0x000c, serial=E6616408432C1933
Info : CMSIS-DAP: SWD supported
Info : CMSIS-DAP: Atomic commands supported
Info : CMSIS-DAP: Test domain timer supported
Info : CMSIS-DAP: FW Version = 2.0.0
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : SWCLK/TCK = 0 SWDIO/TMS = 0 TDI = 0 TDO = 0 nTRST = 0 nRESET = 0
Info : CMSIS-DAP: Interface ready
Info : clock speed 5000 kHz
Info : SWD DPIDR 0x0bc12477, DLPIDR 0x00000001
in procedure 'program'
** OpenOCD init failed **
shutdown command invoked

And FWIW, here's the same failure in the hello_dormant project: 

$ pwd
~/pico/pico-playground/build/sleep/hello_dormant 

$ sudo openocd -d2 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program hello_dormant.elf verify reset exit"
Open On-Chip Debugger 0.12.0-g4d87f6d (2024-02-11-10:09)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
debug_level: 2

Info : Hardware thread awareness created
Info : Hardware thread awareness created
adapter speed: 5000 kHz

Info : Using CMSIS-DAPv2 interface with VID:PID=0x2e8a:0x000c, serial=E6616408432C1933
Info : CMSIS-DAP: SWD supported
Info : CMSIS-DAP: Atomic commands supported
Info : CMSIS-DAP: Test domain timer supported
Info : CMSIS-DAP: FW Version = 2.0.0
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : SWCLK/TCK = 0 SWDIO/TMS = 0 TDI = 0 TDO = 0 nTRST = 0 nRESET = 0
Info : CMSIS-DAP: Interface ready
Info : clock speed 5000 kHz
Info : SWD DPIDR 0x0bc12477, DLPIDR 0x00000001
Error: Failed to connect multidrop rp2040.dap1
Warn : Connecting DP: stalled AP operation, issuing ABORT
in procedure 'program'
** OpenOCD init failed **
shutdown command invoked

I could surely use some help!

seamusdemora commented 1 month ago

Perhaps the answer here is this: Re-install your OS on a blank SD card?

ChristianIvarsson commented 1 month ago

Not sure if you're experiencing the same weirdness as I have experienced a couple of times but sometimes, for I have no idea, the rp2040 suddenly decides it's NOT having any of that SWD/JTAG business no matter which adapter or setting I try and the only way to remedy this issue is to do the bootsel mass storage method and drop the firmware image on it that way. It'll then proceed to accept SWD again as if nothing happened and I go "..WHAT?!" 🤣

lurch commented 1 month ago

@ChristianIvarsson I guess if you're very unlucky, you might end up with code in Flash which messes with the PADS_BANK0: SWCLK or PADS_BANK0: SWD registers (see page 301 of https://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdf ) or SYSCFG: DBGFORCE register (see page 307 of the same PDF), which might make the SWD pins unusable until the Flash has been updated with something else or entirely erased (e.g. https://www.raspberrypi.com/documentation/microcontrollers/pico-series.html#resetting-flash-memory ) ? :shrug:

ChristianIvarsson commented 1 month ago

@lurch It does respond to the ID request when it happens so the port is probably working? It's as if the core has just crashed in some really bad way that prevents it from responding to register and memory requests.

Over the three years or so of tinkering with these, I believe it has happened no more than a handful of times so it's definitely just me doing something it really didn't like or some random tech be techin' moments.

OOCD does, however spit out the same nonsense message when it happens so I were just checking if he's perhaps been bitten by the same tech gremlin as me 😄

seamusdemora commented 1 month ago

@ChristianIvarsson

Chris - thanks for weighing in... I didn't actually get much help, but the old saying still applies: "Misery loves company." LOL :)

But back to the point of discussion: I actually did try the bootsel mass storage method, but that does not seem to work either!! If it weren't for the fact that I can still mount the Pico's USB drive, I would probably guess the entire board has failed.

In my case it seems this has been a rolling train wreck that began with the upgrade last month. I suppose I should just be grateful that it worked for several months prior to that. My current plan is to build a new RPi 5 system from scratch, and re-install the SDK from scratch. I've ordered a new "Debug Probe", and a new Pico board - only because I've got time invested in this.

I'll follow up here if I learn anything, and please do the same if you hear of a magic spell for OOCD.

seamusdemora commented 1 month ago

@lurch : Surprised to see you here... I assumed you recommended I post here because there were some knowledgeable folks that worked this repo.

seamusdemora commented 1 month ago

Update, Sep 29-Oct 1:

I've just replaced the Pico board and the Debug Probe with new, fresh-out-of-the-box hardware.

First, I tried to load my INA260 code which has been running successfully - until recently - since February 2024:

$ sudo openocd -d0 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program ina260_trial.elf verify reset exit"
Open On-Chip Debugger 0.12.0-g4d87f6d (2024-02-11-10:09)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
debug_level: 0

adapter speed: 5000 kHz

[rp2040.core0] halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
[rp2040.core1] halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
** Programming Started **
** Programming Finished **
** Verify Started **
** Verified OK **
** Resetting Target **
shutdown command invoked

Which was SUCCESSFUL, and I got output from minicom (but on TTYACM1?) Then, I tried to load the (slightly) modified hello_dormant code from ~/pico/pico-playground/build/sleep/hello_dormant

sudo openocd -d2 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program hello_dormant.elf verify reset exit"
Open On-Chip Debugger 0.12.0-g4d87f6d (2024-02-11-10:09)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
debug_level: 2

Info : Hardware thread awareness created
Info : Hardware thread awareness created
adapter speed: 5000 kHz

Info : Using CMSIS-DAPv2 interface with VID:PID=0x2e8a:0x000c, serial=E6633861A385812C
Info : CMSIS-DAP: SWD supported
Info : CMSIS-DAP: Atomic commands supported
Info : CMSIS-DAP: Test domain timer supported
Info : CMSIS-DAP: FW Version = 2.0.0
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : SWCLK/TCK = 0 SWDIO/TMS = 0 TDI = 0 TDO = 0 nTRST = 0 nRESET = 0
Info : CMSIS-DAP: Interface ready
Info : clock speed 5000 kHz
Info : SWD DPIDR 0x0bc12477, DLPIDR 0x00000001
Info : SWD DPIDR 0x0bc12477, DLPIDR 0x10000001
Info : [rp2040.core0] Cortex-M0+ r0p1 processor detected
Info : [rp2040.core0] target has 4 breakpoints, 2 watchpoints
Info : [rp2040.core1] Cortex-M0+ r0p1 processor detected
Info : [rp2040.core1] target has 4 breakpoints, 2 watchpoints
Info : starting gdb server for rp2040.core0 on 3333
Info : Listening on port 3333 for gdb connections
[rp2040.core0] halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
[rp2040.core1] halted due to debug-request, current mode: Thread
xPSR: 0xf1000000 pc: 0x000000ea msp: 0x20041f00
** Programming Started **
Info : Found flash device 'win w25q16jv' (ID 0x001540ef)
Info : RP2040 B0 Flash Probe: 2097152 bytes @0x10000000, in 32 sectors

Info : Padding image section 1 at 0x100049b8 with 72 bytes (bank write end alignment)
Warn : Adding extra erase range, 0x10004a00 .. 0x1000ffff
** Programming Finished **
** Verify Started **
** Verified OK **
** Resetting Target **
shutdown command invoked

Which was *apparently successful, BUT I got no output in minicom??

I tried the INA260 project again - to verify it was still working - but it failed: 

$ sudo openocd -d0 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program ina260_trial.elf verify reset exit"
Open On-Chip Debugger 0.12.0-g4d87f6d (2024-02-11-10:09)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
debug_level: 0

adapter speed: 5000 kHz

Error: Failed to connect multidrop rp2040.dap0
Error: Failed to connect multidrop rp2040.dap0
Error: [rp2040.core0] Could not find MEM-AP to control the core
Error: Failed to connect multidrop rp2040.dap1
Error: Failed to connect multidrop rp2040.dap1
Error: [rp2040.core1] Could not find MEM-AP to control the core
Error: Failed to connect multidrop rp2040.dap0
Error: Failed to connect multidrop rp2040.dap0
Error: [rp2040.core0] Could not find MEM-AP to control the core
Error: Failed to connect multidrop rp2040.dap1
Error: Failed to connect multidrop rp2040.dap1
Error: [rp2040.core1] Could not find MEM-AP to control the core
Error: [rp2040.core0] Debug AP not available, reset NOT asserted!
** Unable to reset target **
shutdown command invoked

I pulled the USB power cord from the Pico (to reset it), and tried INA260 again: 

$ sudo openocd -d0 -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 5000" -c "program ina260_trial.elf verify reset exit"
Open On-Chip Debugger 0.12.0-g4d87f6d (2024-02-11-10:09)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
debug_level: 0

adapter speed: 5000 kHz

in procedure 'program'
** OpenOCD init failed **
shutdown command invoked

So I am back to where I started before replacing the Pico and the Debug Probe.

Any ideas would be appreciated.