Inconsistent flashing on a Raspberry Pi Pico W

[znmeb]

I have a Fedora Linux system running PlatformIO and I'm testing a build against a Raspberry Pi Pico W. I just ran the same job 16 times and sometimes the upload fails and sometimes it succeeds:

❯ grep "^test_pico_w  " test_logs/* 
test_logs/test1.log:test_pico_w          FAILED    00:01:15.686
test_logs/test2.log:test_pico_w          SUCCESS   00:00:12.183
test_logs/test3.log:test_pico_w          FAILED    00:00:03.853
test_logs/test4.log:test_pico_w          SUCCESS   00:00:12.232
test_logs/test5.log:test_pico_w          SUCCESS   00:00:09.955
test_logs/test6.log:test_pico_w          FAILED    00:00:03.808
test_logs/test7.log:test_pico_w          SUCCESS   00:00:12.161
test_logs/test8.log:test_pico_w          SUCCESS   00:00:09.946
test_logs/test9.log:test_pico_w          FAILED    00:00:03.434
test_logs/testa.log:test_pico_w          SUCCESS   00:00:12.197
test_logs/testb.log:test_pico_w          SUCCESS   00:00:09.971
test_logs/testc.log:test_pico_w          FAILED    00:00:03.472
test_logs/testd.log:test_pico_w          SUCCESS   00:00:12.064
test_logs/teste.log:test_pico_w          FAILED    00:00:03.876
test_logs/testf.log:test_pico_w          SUCCESS   00:00:12.146
test_logs/testg.log:test_pico_w          SUCCESS   00:00:10.051

I've captured verbose logs of all the tests in this zipfile: test_logs.zip

This is the lsusb for the Pico:

Bus 001 Device 069: ID 2e8a:000a Raspberry Pi Pico W

and the serial port:

❯ ls -l /dev/ttyACM0 
crw-rw-rw-. 1 root dialout 166, 0 Sep  6 20:13 /dev/ttyACM0

I looked at some of the log files and it looks like when it fails, it correctly figures out that the board is at /dev/ttyACM0, tries to reset it, thinks it did so and proceeds with the upload, which then fails. But when it succeeds, it sometimes thinks the board is at /dev/ttyS31 and resets that, which wouldn't touch the actual board. Then it uses picotool to upload the uf2, which succeeds because picotool knows where the board it.

Here's the platformio.ini entry for the board:

[env:test_pico_w]
extends = target
platform = https://github.com/maxgerhardt/platform-raspberrypi.git
board = rpipicow
framework = arduino
board_build.core = earlephilhower
build_flags = ${target.target_build_flags} -DMAXDICT=0x2C000
build_type = debug

[maxgerhardt]

It appears to me that there might be a timing issue: In cases where it fails it tries to reset /dev/ttyACM0 (which looks correct) but then invokes picotool too fast, the board might not have rebooted into BOOTSEL mode properly yet. In cases where the upload works, the board has been rebooted into picotool before quite some time, so upload works. I've already noted that before when using picotool to upload filesystem data and added a generic 0.5 second delay. But I neglected to do that when I switched the uploader from the outdated rp2040load (no RP2350 support) to picotool directly. Let me just try something.

[maxgerhardt]

Do a pio pkg update -g -p raspberrypi, I have commited 19f500fe2c7c8603f313972a3ca28a3bb1ff006b, and redo your tests.

[znmeb]

Thanks! I'll try it. Meanwhile, I have captured Linux journalclt logs of a successful test and a failed one. If it still fails after I do the update I'll capture new logs and upload them.

[maxgerhardt]

Actually, I've gone ahead and commited a method that should be even better: Wait until picotool info -d reports a connected RPxxxx device. That's done in https://github.com/maxgerhardt/platform-raspberrypi/commit/6cd32488770b19533b474bea4b8de3cbc57237fe. So, please do your test with the latest latest version.

[znmeb]

Done - test log and accompanying journalctl in attached zipfile. I think you can close this!

retest.zip

[maxgerhardt]

The logs show the case where the pico was already in BOOTSEL mode. It ran into the max timeout of 3.0 seconds there until invoking picotool load. Can you run that just again to see if the "device runs regular Arduino sketch and gets rebooted into BOOTSELF mode" case works too?

[znmeb]

OK ... by the way, the one I just ran isn't getting the devices right. I don't know how it found the Pico on ttyS31 when it's actually on ttyACM0. Is that something known inside the software (yours or Earle's)?

[maxgerhardt]

If the device is in BOOTSEL mode, it does not create a /dev/ttyXXX. Then, the PlatformIO core has some fallback code to get some COM device. It should not be the case that you have /dev/ttyACM0 visible on your machine as well as PlatformIO picking /dev/ttyS31.

[znmeb]

If the device is in BOOTSEL mode, it does not create a /dev/ttyXXX. Then, the PlatformIO core has some fallback code to get some COM device. It should not be the case that you have /dev/ttyACM0 visible on your machine as well as PlatformIO picking /dev/ttyS31.

I think I saw that case - /dev/ttyACM0 visible - in the previous failed runs yesterday. It found /dev/ttyACM0, sent it a reset, but didn't wait for the BOOTSEL mode to happen.

[znmeb]

Here's the new test with BOOTSELenabled going in:

manual-bootsel-updated-pio.zip

Is there a way it can test for BOOTSELmode and just go ahead and upload with picotool?

I did a lot of testing with picotoola week or so ago. It looks like there are three possible responses from picotool info -a:

1. If the device is in BOOTSEL mode, it will go ahead and return the state and anything it finds in the flash.
2. If the device is connected to a USB serial TTY, it will list something like this:

No accessible RP2040/RP2350 devices in BOOTSEL mode were found.

    but:

Device at bus 1, address 16 appears to be a RP2040 device with a USB serial connection, so consider -f (or -F) to force reboot in order to run the command.

3. If it can't find the device, it will say this and you need to manually put it in BOOTSEL mode:

No accessible RP2040/RP2350 devices in BOOTSEL mode were found.

P.S.: When I just ran that, the device was a SparkFun Pro Micro RP2350. So it can't tell whether it's an RP2040 or an RP2350 unless it's in BOOTSEL mode.

[maxgerhardt]

Try again: I've commited f4f43a2ef491545f0579ace5142b8d214e0fc15a, and there's now a fastpath if the device is already in BOOTSEL mode. This really should only ever happen once though, since once it has an Arduino firmware loaded into it, it should always do the 1200bps reset.

[znmeb]

Looks like everything on your end is working. bootsel_tests.zip

For some reason, once PlatformIO successfully flashes the Pico W, PlatformIO's reset code works and I can re-flash it, but the host picotool loses the ability to reset the Pico W:

❯ picotool info --all
No accessible RP2040/RP2350 devices in BOOTSEL mode were found.

but:

Device at bus 1, address 69 appears to be a RP2040 device with a USB serial connection, so consider -f (or -F)
    to force reboot in order to run the command.

cforth/RP2350-testing on  master 
❯ picotool info --all --bus 1 --address 69 --force
Tracking device serial number E66130100F5C9C34 for reboot
ERROR: Unable to locate reset interface on the device

cforth/RP2350-testing on  master 

Here's what picotool finds if I put the Pico W in BOOTSEL mode manually:

❯ picotool info --all --bus 1 --address 71
Program Information
 none

Fixed Pin Information
 none

Build Information
 none

Device Information
 type:        RP2040
 revision:    B2
 flash size:  2048K
 flash id:    0xE66130100F5C9C34

cforth/RP2350-testing on  master

If I put the Pico W in manual BOOTSEL mode and re-flash it with the SDK hello_usb example, the host picotool finds it as always. It also found it in my previous tests with PlatformIO, I think. So I don't know if it's something missing in the source code for these tests that the SDK example has, something PlatformIO knows how to do that picotool doesn't, or something else.

[znmeb]

Looks like everything on your end is working. bootsel_tests.zip

For some reason, once PlatformIO successfully flashes the Pico W, PlatformIO's reset code works and I can re-flash it, but the host picotool loses the ability to reset the Pico W:

❯ picotool info --all
No accessible RP2040/RP2350 devices in BOOTSEL mode were found.

but:

Device at bus 1, address 69 appears to be a RP2040 device with a USB serial connection, so consider -f (or -F)
    to force reboot in order to run the command.

cforth/RP2350-testing on  master 
❯ picotool info --all --bus 1 --address 69 --force
Tracking device serial number E66130100F5C9C34 for reboot
ERROR: Unable to locate reset interface on the device

cforth/RP2350-testing on  master 

Here's what picotool finds if I put the Pico W in BOOTSEL mode manually:

❯ picotool info --all --bus 1 --address 71
Program Information
 none

Fixed Pin Information
 none

Build Information
 none

Device Information
 type:        RP2040
 revision:    B2
 flash size:  2048K
 flash id:    0xE66130100F5C9C34

cforth/RP2350-testing on  master

If I put the Pico W in manual BOOTSEL mode and re-flash it with the SDK hello_usb example, the host picotool finds it as always. It also found it in my previous tests with PlatformIO, I think. So I don't know if it's something missing in the source code for these tests that the SDK example has, something PlatformIO knows how to do that picotool doesn't, or something else.

[maxgerhardt]

If I put the Pico W in manual BOOTSEL mode and re-flash it with the SDK hello_usb example, the host picotool finds it as always. It also found it in my previous tests with PlatformIO, I think. So I don't know if it's something missing in the source code for these tests that the SDK example has, something PlatformIO knows how to do that picotool doesn't, or something else.

Interesting! I did not expect this. When an Arduino-framework code is loaded, then picotool never recognized anything for me anymore. Which is understandable since the UF2 bootloader isn't running on the device right now. I always have to do the 1200bps reset for picotool to work with the device at all. But if you say, that flashing hello_usb results in a device that picotool can still recognize, reset and reflash at will, that's interesting..

[znmeb]

I'm going to call this a picotool bug for now, since PlatformIO can flash the device. I just tested it on the SparkFun Pro Micro RP2350 to rule out a defective board or some electronic issue with the Waveshare eval board the Pico W is plugged into. Picotool can't see the SparkFun either after flashing with PlatformIO, but PlatformIO can flash the board again, and whether picotool can reset a board shouldn't depend on what firmware is currently running in the board.

I'm going to see if I can come up with a simpler test to see if I can reproduce this using the Pico SDK to build and flash the SparkFun Pro Micro RP2350. That will make it easier to open an issue against picotool. You can close this issue if you wish. Please let me know if you have any other tests you want me to run.

[znmeb]

Apparently this is a known issue but it's not clear to me what has to be changed - picotool or PlatformIO. I am building picotool from source so if it can be done at compile time I'll test it out.

https://github.com/raspberrypi/picotool/issues/88

[maxgerhardt]

Mhm. Maybe we'll revisit this later when this platform can fully build Pico-SDK firmwares to confirm the issue. The issue with inconsistent flashing seems to be properly resolved now. Reset is done via 1200bps touch, not via picotool in any case.