STM32 processors cannot receive UART at even moderate baudrates while asleep

multiplemonomials commented 6 months ago

@Ky-Ng and I were investigating a mysterious issue where an Mbed processor was dropping bytes sent to it by a Python script over a serial port (at 115200 baud). We were eventually able to localize the issue to this old Mbed bug: https://github.com/ARMmbed/mbed-os/issues/8714. As it turns out, this is quite a serious issue.

It's a long thread, but to summarize:

Most STM32 processors do not have a UART Rx FIFO
This means that the processor must always respond to a UART interrupt in well under one byte-reception time (which is approximately (10/baudrate)), or data will be lost
Some mitigations were implemented, such as locking deep sleep in BufferedSerial and improving the LP ticker code to run at lower ISR priority and wake up quicker
However, the time to wake up from sleep is still long enough to cause problems -- this guy measured it at around the 25-80us range
This means that even moderately high baudrates (57600 or 115200 baud) can send bytes fast enough that the MCU will lose data.

This is a kinda difficult issue, and a case can be made that it's actually the STMicro hardware's fault for not including a FIFO buffer in the UART, something that I have never seen any MCU do before. And, to their credit, they did actually fix this, but only in their LPUART peripheral and in some of their newer processors (G0, G4, L5, U5, WB, WL). To make matters worse, even when the FIFO does exist in hardware, Mbed's TARGET_STM/serial_api.c isn't capable of actually using it! So this issue currently affects all STMicro chips, even the ones that wouldn't, in theory, be subject to the problem.

Conditions to Reproduce

This issue shows up under the following conditions:

Using an STMicro processor of any kind
UART baudrate is set to greater than roughly 57600 (though the threshold could be even lower on slower MCUs like the STM32WB, or if the core clock is at a lower value)
The MCU is in sleep mode.
Multiple bytes are received in rapid succession during the time that the MCU is asleep.

Under these conditions, on an STM32L4 at 115200 baud, we observed that second byte sent by the PC would almost always be lost, and the firmware would only receive the 1st and 3rd bytes transmitted.

Note that this is easier to reproduce in Mbed CE than in Mbed OS, because we changed the default baudrate to 115200, and we also recommend that projects use buffered serial by default. Buffered serial is blocking by default, meaning that if you do a scanf() which scans multiple bytes from the console, and then send those bytes via a script (as in, not typing them by hand), that will create the necessary conditions for this problem to appear.

Also note that the following options do not cause or fix the issue, but might change the exact baudrate where it happens (making it appear to show up or go away):

Enabling/disabling tickless mode
target.lpticker_delay_ticks option
target.tickless-from-us-ticker option
Choice of BufferedSerial vs UnbufferedSerial

Current Workarounds

Reducing the Baudrate

If you need sleeping, but can tolerate a slow console, the easy solution is to just slow down the serial port. 9600 baud should be OK, because that allows about 1ms for the processor to process each character. Just drop the following in your mbed_app.json:

"platform.stdio-baud-rate": 9600,

and that should sort out the issue (though it will also take your serial speeds back to the 90s).

Disabling Sleep

If you want a fast console, but don't need sleeping, you can also easily work around the issue by disabling sleep entirely. To do that, you must create a custom target for your board, and add a block like

"device_has_remove": ["SLEEP"]

to its custom_targets.json file. This prevents the MCU from ever going to sleep, ensuring that it will be awake and alert whenever something sends it UART data. However, it is, of course, totally inappropriate for any applications which rely on low power operation.

Also note that if something else locks interrupts (via a critical section) for 10s of microseconds, the issue could still show up. So, I would advise using a little bit of caution with high baudrates even with this workaround, as there could be other places in first or third party code that are potential problem areas -- an exhaustive test hasn't been done.

Future Fixes

Making Use of the UART FIFO

If the Mbed drivers could be updated to use the UART peripheral's buffer, that could be the cleanest fix for the majority of devices -- the majority of STM32 chips either have a LPUART available or have UARTs with FIFOs. We just have to make sure that PCB designers know to prioritize the LPUARTs for any application that needs to receive at high-ish baudrates...

Reducing Default Baudrate

It would be fairly easy to reduce the baudrate for STM32 devices down to 9600 baud in targets.json5. However, I really don't like the idea of doing this, because is isn't intuitive for users why some devices would run at different baudrates than others. Plus, 9600 baud is just so damn slow... I hate the idea that we'd be using a serial port at 1980s speeds.

Printing a Warning

Another easy fix would be to update Mbed to print a warning at runtime if all 4 of the following conditions are true:

Running on an STMicro chip with no UART buffer supported by the software
Sleep is enabled
Serial Rx is enabled
And, the baudrate is set above 9600

This way, users would be directed to either reduce the baudrate, turn off sleep, or disable serial Rx.

Using DMA

The best and most universal fix for this issue would be to use DMA to empty the UART Rx buffer instead of relying on an interrupt. This way, the buffer could be emptied as soon as the DMA controller is powered back on (if it even gets put to sleep at all!). This is definitely supported by the hardware, and we have at least some of the software infrastructure implemented now thanks to my prior work on SPI DMA. However, as @kjbracey mentioned here, using DMA for UART can be tough because you might want to do a circular buffer, and/or you need to carefully monitor how many characters the DMA has actually written in the background.

wdx04 commented 3 months ago

Hi, I just created a UART DMA driver for STM32 processors, you may check it out: https://github.com/wdx04/BurstSerial

multiplemonomials commented 3 months ago

This looks neat! Will have to take a deeper look sometime. By the way, you might be interested in this header: https://github.com/mbed-ce/mbed-os/blob/master/targets/TARGET_STM/stm_dma_utils.h

It handles all the family-specific parts of setting up a DMA transaction and should let you remove the burden of implementing the ISR handlers correctly for each family (which was quite difficult to get right!)

multiplemonomials commented 3 months ago

By the way, I would definitely accept PRs for adding this (or other UART functionality like enabling the FIFOs) into Mbed!

wdx04 commented 3 months ago

By the way, I would definitely accept PRs for adding this (or other UART functionality like enabling the FIFOs) into Mbed!

OK, I'll consider submitting a PR later. By the way, do you have any plans to merge with the official Mbed OS repository? There are no commits in 4 months in the official Mbed OS repository, I think it's a good time to merge the code and make Mbed CE a superset of the official Mbed.

multiplemonomials commented 3 months ago

Oh like, cherry pick the remaining changes from Mbed master? Yeah someone needs to do that, I'd accept a PR! The last time we did this was https://github.com/mbed-ce/mbed-os/pull/185

mbed-ce / mbed-os