Custom probe breaks on second boot

TimGoll commented 5 days ago

We designed a custom pico debug probe with a design very similar to the original debug probe. The design can be seen in the attached schematic: LibroPicoProbe.pdf

We ordered them from JLC about a year ago (09.23) and all of them worked just fine (we flashed the firmware from this repository via USB). Recently some of these boards stared dying. We first assumed that this was a hardware defect, but then investigated things further. We are still able to flash the board via USB and via SWD. After flashing it, the board automatically starts the flashed firmware and the firmware works without any issues.

However, in both cases a power cycle breaks the board and it fails to start up. Later tests with probe-rs (gdb based) showed that the memory address returned by the step command is within reasonable ranges for the first power up after the flashing process. But for all subsequent attempts, it is 0xfffffffe, which seems to be some kind of error state.

Our research hinted that this could be due to bad oscillators. This seems unlikely because these boards worked at some point, work in the initial run and the other boards from the same batch work as well. However we still tested it with two other programs. A small blinky and a multi megabyte project done by us. The first being smaller than the flasher binary, the second being larger. Both use the external crystal and both work fine. On the same hardware where the debugprobe binary breaks.

At this point we're at loss. We can only reproduce this issue with this specific binary, everything else seems to work well. Additionally we are using the same RP2040-flash-quartz combo since 2023 in multiple projects without any issues so far.

One thing that we want to verify is that the flash is not broken. As soon as we receive new flash ICs, I will desolder them from the "broken" boards and replace them with the new ones. I doubt this will fix anything, but it is worth a try.

Maybe it is something obvious that we are missing, I'm not sure. Maybe someone else has a similar problem, I at least found a few posts in some forums without any solutions:

UF2 Bootloader v3.0
Model: Raspberry Pi RP2
Board-ID: RPI-RP2

lurch commented 2 days ago

Our research hinted that this could be due to bad oscillators.

You might want to try creating a custom header file for your board, and experiment with different values for PICO_XOSC_STARTUP_DELAY_MULTIPLIER? See e.g. https://github.com/raspberrypi/pico-sdk/pull/457 and the various other issues that it links to.

TimGoll commented 2 days ago

Our research hinted that this could be due to bad oscillators.

You might want to try creating a custom header file for your board, and experiment with different values for PICO_XOSC_STARTUP_DELAY_MULTIPLIER? See e.g. raspberrypi/pico-sdk#457 and the various other issues that it links to.

That's worth a try. Thank you. I do wonder though why the problem only occurred after a year of use. Maybe some degradation of the MLCCs or the crystal over time?

raspberrypi / debugprobe

Custom probe breaks on second boot #154