USB (Serial) not working in with Dualcore

[savejeff]

It took me a while to notice this as I tested everything with the pico probe but when i use the second core the USB serial stops working. i tested the Mbed core with single core as well as the arduino-pico core with only loop and not loop1 and both work without a problem

Where does the USB Driver run? Is the USB Code just not executed as both cores are working on user code? Is there a way to execute something like a USB.maintain() function?

The bevahiour I'm getting when the USB is not working is that on connect wit a serial console (under windows and linux) the interface freezes and timeouts after around 30s.

i tried to update the platform package and it updated the TinyUSB lib. sadly no improvement.

[earlephilhower]

Sorry, that isn't happening to me or other folks as far as I know. See my 1st multicore example https://github.com/earlephilhower/arduino-pico/blob/master/libraries/rp2040/examples/Multicore/Multicore.ino which is all serial port outputs. We've also done FreeRTOS debug using printouts over all cores.

Can you post a small MCVE showing the problem?

One thing you may want to check is that Windows or Linux may change the serial port used for the chip after upload, because as far as they know the USB port was unplugged during the upload. So, make sure you're connected to the right (new) port after upload.

[savejeff]

Thx for the quick response.

I think this is little bit more complex problem. Ill try to reduce the problem to a MCVE.

After the core update, it seems to work fine if i start/power up the board with the USB connection. When the board is already supplied through another power source, and then plugged in USB, the USB is not recognized. It takes a while (around 15s) before i hear the sound that a new USB device was plugged in. But there is no new COM port. this behaviour happens only with the dual core version. single core allows hotplug while running.

I'll test the multicore.ino example tomorrow. the problem might only appear with actual load on both cores not only print and delay.

EDIT: okay i tested the multicore example and it works with hotplug. ill start to work on a MCVE tomorrow. this is something more complex

[earlephilhower]

There have been reports of the 2nd core not properly coming out of reset when using Picoprobe. The same code runs properly on a cold boot or on a warm boot outside of using the SWD channel, so there may be some oddities in the OpenOCD profile. Symptoms here would be the 2nd core just never starts, not that the Serial port doesn't work from it, though.

[savejeff]

Its nothing with the SWD or Picoprobe.

I only use the USB connection to upload and connect. I narrowed it down to my Software I2C Library. USB stops working when i run the software I2C on the second core. Im still trying to produce a MCVE that require no or almost no special hardware. Do you my any change have a SSH1303 OLED display around? its a pretty common module

[earlephilhower]

I will check through my junk box but I don't think I have that kind of display.

I would suggest at this point to connect to the board using a Picoprobe and GDB. Check that thread 1 and thread 2 are not crashed. The way the Pico works is you can have a hard fault on one core (i.e. reading from NULL) but the other core can continue just fine. So, if one or both cores are crashed then the USB port will obviously be dead, too, since it's run in software.

[savejeff]

I have something like a MCVE. Ive reduced my project to a level here i don't leak to much code. reducing it more would make it harder to switch between single and dual core in the way it currently works.

Could you test this on your system? The code does not crash, even when USB freezes on the host. The project does not need an actual Oled display attached, as it only has writes and no reads in the i2c protocol. my software i2c implementation ignores missing ACKs.

What i found out is: it (most likely) only produced the error if on the first core the Flash EEPROM is used and on the second core i2c communication with the display is running concurrently. but i also have produced the USB freeze in the single core version

The best way to produce the error is to disconnect and reconnect to the serial port / com port every some seconds with the pico stays connected and powered on. after 1 to 5 reconnects the USB freezes and i have to restart the serial monitor and disconnect the USB. sometimes even restart the pc.

You can switch between single and dual core code through the different build configurations.

here the PlatformIO project: Demo_RP2040_Multicore_USB_Freez.zip

i tested this on a plane Raspberry Pi Pico without any additional hardware attached. The Display Library "ssd1306" is a pretty commonly used library for the display and should not be the root of the problem, but its the only part of my code that uses C not C++ code.

[earlephilhower]

Wow, that's an MCVE alright...a Macro Complete Verifiable Example. 😆 I'll try and see what I can do, but no promises as it's ginormous and I need to port it to standard IDE since I don't run P.IO.

What i found out is: it (most likely) only produced the error if on the first core the Flash EEPROM is used and on the second core i2c communication with the display is running concurrently. but i also have produced the USB freeze in the single core version

An important thing to note is that to actually write the flash (EEPROM.commit(), not EEPROM.write() which just copies to a RAM buffer) is that it needs to freeze the other core via an IRQ from the x-core FIFO. If you have disabled interrupts or done something funky like use the HW exceptions to swap tasks, you will end up crashing the core that's not writing to flash.

The best way to produce the error is to disconnect and reconnect to the serial port / com port every some seconds with the pico stays connected and powered on. after 1 to 5 reconnects the USB freezes and i have to restart the serial monitor and disconnect the USB. sometimes even restart the pc.

Do you mean just starting/stopping minicom or putty serial monitors, or do you mean physical plug/unplug events?

[savejeff]

Haha sry, this was the result of 5h of throwing out code and more code to reduce it to the minimum sceleton that still demonstrates the problem. the project is cross platform and runs on RP2040, SAMD, ESP32 and Teensy with a lot of modules for everything from GNSS, IMU, LTE, WIFI, BLE, Ethernet, SD Card etc. ^^. About PIO, i recommend just installing it as the project compiles and runs without external dependencies, its a complete package. It will be way harder to port this to Arduino IDE, especially as i use compiler flags to enable architecture/platform specific code segments (its also possible to switch to the mbed rp2040 core). With PIO you basically just have to open the project, select dual or single core compilation configuration and hit run. no library installation, no other setup. debug with pico probe can be done with F5 using the picoprobe compile configuration.

In the given "MCVE" i use the EEPROM but inter-buffer it with RAM. so only Core0 interacts with EEPROM itself. the other core just reads the copy held in RAM. this is also secured with a MUTEX to avoid simultaneous writes. The code can be found in MemoryModule

It is enough to just close and open the putty/serial monitor connection. physically the board should/must stay connected to avoid reboot. i use hterm and just toggle connect/disconnect every couple of seconds and it freezes after one or two connects. Before i reduced the code it was even unable to connect a single time in dual core mode, but its gotten better with less code

The first thing would be to see if you can reproduce this on your end. if that is the case I'll try to reduce the code even more. i would also like to try a different Display library, but I2C interactions (with a library that i know to be very reliable for many years now) should not be the cause of this behaviour

[earlephilhower]

While waiting for me to find a good time to look into this, if you have a PicoProbe setup, could you run GDB when the Serial is hung and do

thread 1
where
thread 2
where

(I think it's thread 1 and thread 2, but if thread 2 fails then try thread 0 and thread 1)

That will determine if, say, one core is stuck in a mutex wait or has crashed. That kind of info really helps point to the culprit.

[savejeff]

Sorry, i was busy with a little side project over the weekend. I started an Arduino Core for Raspberry Pi in PlatformIO. I've already got GPIO, I2C, SPI, Serial, and EEPROM working ^^. No Idea why this was not done earlier.

About the Debugging: I already checked the state of the cores via PicoProbe in VSC (which uses GDB i think). Both Threads/Cores run with no crashes. My User code with runs on both cores work fine without lag or halts. I don't think the RP2040 driver is aware of the freeze, it's more like it does not respond to incoming events from USB (just a feeling).

I can check again to make sure later.

[earlephilhower]

I really can't get the massive example you posted to build. It's redefining core internal values and the jXXX libs do not seem to build properly for me.

If you can come up with a real MCVE we can re-open, but with this full app it's just not possible.

[savejeff]

Have you tried to compile it with PlatformIO ? As I said, the code is not easily ported over to the Arduino IDE as i use some build flags. With platformIO it should be as easy as opening and compiling the project without additional setup

Before investing more time into producing an MCVE, which might take multiple days of work, i wanted to check that this problem is reproducible at your end.

Also, it seems the USB freezes get less and less frequent the more code i remove from the project. Where I started of the project produced the freeze every single try. the reduced version only produces is about every 5-10 tries. If this is a more complex problem that only occurs when enough varied work is done where the USB code does not get enough time to work, it will be hard to reduce to a single page of code.

[savejeff]

just to finish this cleanly: a recent update seems to have fixed the problem. There seems to be no freezes of USB Serial in dual-core mode.