Noisy UART at higher baud rates

[seankovacs]

Board

ESP32-C3/C6-M1

Device Description

I have an ESP32-C3-M1 and ESP32-C6-M1 that attach to an STM32 via UART.

Hardware Configuration

No

Version

latest master (checkout manually)

IDE Name

Platformio & ESP-IDF w/ Arduino

Operating System

Windows 10

Flash frequency

40

PSRAM enabled

yes

Upload speed

115200

Description

Compared to the same code running on an ESP8285 (Crystal @ 26Mhz), both ESP32 eval boards have increased CRC errors at baud rates higher than 460800. The ESP8285 board has no issue transmitting over the same lines/setup with nearly 0 CRC errors @ 921600 baud with the STM32. The ESP32-C3 eval uses the exact same code as the ESP8285 - that is, PlatformIO. The ESP32-C6 variant using mostly the same code, just outside of Platformio with ESP-IDF with the Arduino component. Both ESP32 boards transfer a lot of garbage at that higher baud rate and I'm not sure why. I've found other people describe very similar problems when the config CONFIG_PM_ENABLE is set, but that isn't set for me. Are there other settings and/or code changes I can test to see if he noise level/crc errors drop?

Sketch

This is the repo/code I'm running (for C3/8266): https://github.com/skybrush-io/mavesp8266/tree/feat/esp32

I have a local repo that compiles against my C6 board that is roughly the same code.

Debug Message

N/A

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[Jason2866]

esp8266 serial is a software implementation. esp32 has hardware serial. In a perfect world code to use both serial variants should be the same. Saying this, we encountered issues here in project Tasmota too. We had to adjust Tasmota code to work with both implementations. You can't have used latest Arduino master as component from Arduino with Platformio since this is not supported. If you do Platformio will use Arduino core 2.0.14 and IDF 4.4.6

[seankovacs]

Correct - I'm only using the latest 5.x with my ESP-IDF implementation on the C6. The C3 uses IDF4.x and 8266 even lower. :)

Didn't realize it was software vs hardware...I would expect the opposite of what's happening to occur even more than (8266 able to run at slower baud rates compared esp32). I'll keep tinkering with different settings, digging into the code to see if the implementation of the software serial handles things differenly than hardware. It is nice to know there is a difference in the code paths (hard/soft) - helps reason about this a bit more.

[TD-er]

As far as I know, the ESP8266 does have a hardware serial port. However, there is a software layer to allow for buffers larger than the hardware buffer of 128 bytes.

On ESP32 there is also a hardware buffer of 128 byte per HW serial port (3 on ESP32, 2 on ESP32-C3 if I'm not mistaken) However there is some more event handling taking place on the ESP32-variants to keep buffers filled (or read). You can set the threshold for when such events will be triggered. So maybe at really high baudrates you may be run into some unpredictable delays due to RTOS also scheduling other tasks which may cause the bytes being sent at a more irregular interval when the buffer needs to be filled again. At 921600 baud, you are transferring roughly 100kByte/sec. Thus the HW serial buffer of 128 bytes only lasts about 1 msec. Especially on single-core chips handling WiFi will likely take longer than 1 msec to handle making it really hard to keep up transfers at this speed.

You could change the threshold for "buffer empty" events to about 64 byte, so the HW serial buffer will be refilled more frequently and hopefully you can keep up with the high baudrate. Don't forget to also look at increasing the buffers for the serial port. These extra buffers will then be used to fill up the HW serial buffers when they report exceeding the set threshold. This will then be done by Arduino and/or IDF code, so you don't need to change anything else in your own code other than setting the threshold and increasing the buffers.

[seankovacs]

Yeah the ESP8266 does have a HardwareSerial, so that is comparable. I tried a few different buffer size changes, from 0 to really large (1024 * 16) and inbetween. Not really any improvement. I may try and implement a bare bones native ESP-IDF project that has wifi soft AP + udp server + uart <-> udp sending, thus removing Arduino from the mix. Maybe along the way I'll stumble upon the magic setting and/or buffer size that addresses this.

[SuGlider]

@seankovacs - How are the boards connected to each other? Which UART are you using in each board? What is the Arduino Core used in the project?

[SuGlider]

Is there any code to verify how those UART are configured and used?

[seankovacs]

Connected via 100mm jumper wire. I ruled out wiring issue since that varaible is constant across tests. mavesp8266 code uses Serial for UART...I'd imagine that maps to Serial0 for all boards.

In the main Arduino loop() the read message functions are called.

On the STM32/UART side

The serial is initialized like this:

Serial.setRxBufferSize(4096);
Serial.begin(getWorld()->getParameters()->getUartBaudRate());

Data is read like so:

void
MavESP8266Vehicle::readMessageRaw() {
    static uint8_t buf[1024];
    int buf_index = 0;

    while(Serial.available() && buf_index < 300)
    {
        int result = Serial.read();
        if (result >= 0)
        {
            buf[buf_index] = result;
            buf_index++;
        }
    }

    if (buf_index > 0) {
        _forwardTo->sendMessageRaw(buf, buf_index);
    }
}

The writing portion:

int
MavESP8266Vehicle::sendMessageRaw(uint8_t *buffer, int len) {
    Serial.write(buffer, len);
    return len;
}

The code that handles the UDP data (to/from computer)

sendMessageRaw (as seen above in the readMessageRaw method) is a method that sends the data over a UDP socket.

int
MavESP8266GCS::sendMessageRaw(uint8_t *buffer, int len)
{
    _udp.beginPacket(_ip, _udp_port);
    size_t sent = _udp.write(buffer, len);
    _udp.endPacket();
    return sent;
}

And the reading portion:

void
MavESP8266GCS::readMessageRaw() {
    int udp_count = _udp.parsePacket();
    static uint8_t buf[1024];
    int buf_index = 0;

    if(udp_count > 0)
    {
        while(buf_index < udp_count)
        {
            int result = _udp.read();
            if (result >= 0)
            {
                buf[buf_index] = result;
                buf_index++;
            }
        }

        if (buf_index >= 2 && buf[0] == 0x30 && buf[1] == 0x20) {
            // reboot command, switch out of raw mode soon
            getWorld()->getComponent()->resetRawMode();
        }

        _forwardTo->sendMessageRaw((uint8_t*)buf, buf_index);
    }
}

[seankovacs]

After writing out the code, I noticed the strange condition on the loop buf_index < 300. Removing this improved things a lot. Curious why this condition would hurt the ESP32 and not the 8266?

[TD-er]

You could also try adding some resistor in series with the serial wires. Maybe there is some difference in impedance between the ESP8266 and ESP32 or maybe the ESP8266 board already has a roughly 500 Ohm resistor in series to prevent 'rininging'?

I don't know if the ESP32 does do something to the GPIO configuration when it is 'idling', like making the pin 'floating' or perhaps any pull-up resistor internally is different from the ESP8266?

Anyway adding such a resistor in series is never a bad idea and is even mentioned in the design guides. I think they mention a 499 Ohm resistor there, but it isn't that critical. So if you only have 470 Ohm at hand it would also be an improvement over what you're having now.

N.B. Only add it first to the TX line of the ESP32 as the ESP8266 may already have it and the TX on the other end may also have it.

[josef2600]

i don't think they have series resistors for boards. but putting a series resistors to RX and TX line is a must. but not 500 ohm, it's too much. a resistor about 47 to 68 ohm will be good, and add it to both sides. then on one the boards (only), put 10k ohm pull up resistors on RX TX to 5 volt (5 volt is the correct voltage, if you cant, go with 3.3). it will remove the noise from the lines considerably high. do remember the series resistors must be small and put them as near the chips as possible. do not use twisted wires for uart. twisted wires are for rs485 and CAN protocols.

[TD-er]

See: https://www.espressif.com/sites/default/files/documentation/esp8266_hardware_design_guidelines_en.pdf Page 10:

[seankovacs]

I have 200Ohm resistors in series to most digital input & outputs on my STM32 board. I obliged and switched up the series resistors for this UART - both larger and smaller values and saw no change. Even pulling up with 3.3 (can't do 5v) did nothing. Tried only on the TX of the ESP, etc. Looked at both 8266 and ESP32 RX/TX lines on a scope - the lines look fairly clean. Really don't think it's electrical. I'm going to explore any minor differences between the mavlink c library and the older esp-idf used in the 8266 code. I noticed in raw mode (aka transparent mode) it (ESP32) does much better (my comparisons include packets dropped and raw speed ) . It gets very close to the ESP8266 @ 72kBytes/s @ 921600 baud. However, in the default running mode - parsing packets on the ESP - it still drops half the packets on ESP32 ...none of the 8266. At around 460k baud does it start looking like the 8266 in terms of packets dropped atleast (almost 0). I do think I'll eventually call it in terms of messing with this as I'll likely just run at a lower baud rate on the ESP32 which will be fine. I prefer the newer Wifi4/6 on the ESP32(C6) - so if it costs transfer speeds, so be it.

[SuGlider]

mavesp8266 code uses Serial for UART...I'd imagine that maps to Serial0 for all boards.

This is not always true. Serial may be USB CDC also. It depends on the board that is used and also how the board is configured. In order to make sure that the sketch is really using UART, use Serial0 directly in your code. C3 and C6 have native USB HW CDC that may be referenced as Serial by ESP32 Arduino.

[SuGlider]

Another point: UART0 is the console for Log messages. It may break the protocol or communication with Warnings, Errors etc. You may want to use UART1 instead. In devKits, UART0 is also connected to the CP2102 chip, which may attenuate or cause the issues you are seeing.

[SuGlider]

@seankovacs - A suggestion for reading the UART is by using onReceive(), which is only available for HardwareSerial (UART). There is an example: https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/OnReceive_Demo/OnReceive_Demo.ino

This is like setting an "ISR UART function" that is called as soon as data is ready to be read.

[seankovacs]

Good call. I did switch the code to reference Serial1 to get around Serial0 issues with the CP2102 tying up the lines. Yeah - I'll take a look at the onReceive stuff. Looks interesting. Thank you.

[josef2600]

Good call. I did switch the code to reference Serial1 to get around Serial0 issues with the CP2102 tying up the lines. Yeah - I'll take a look at the onReceive stuff. Looks interesting. Thank you.

first of, excellent work by not dropping the work after long term and trying to fix it! but i am a little confused! are you connecting ESP32 to STM directly and checking the data or do you use USB to serial to check it on PC ? these are different conditions. if you are using USB to uart, make sure that device is capable of high speeds, because many of them don't work over 256000 bps. if you are sure the line is electrically clean, then you should go for software. lets say, instead of using the ESP32 hardware uart, turn it off and then use a software serial code. don't make the mistake that thinking the hardware is better! we are working with the chines chips. the main problem of software implementation is the timing. uart is very very depended on timing. but since we have a APB clock over 160 MHz , we shouldn't have a problem of accuracy. but there are 2 other benefit of using hardware. first is hardware interrupts and second one is not getting the code to halt for running. since we actually don't have any interrupt on esp32, except the receiving, it doesn't really matter. and if you use very fast speed, it would be minimal pause. but, for receiving part, you will have a problem! since it has no idea when it is receiving and there is no buffer. at last, i think it should be the problem in driver for uart and do not think we can do anything about it! but there is another solution too! using another micro or use interrupt on the line. with both of them you can have a pin to know when data is on the line, then use interrupt so you can read the data as fast as possible. remember that both RX and TX in idle mode are always positive(+3.3 or +5). so if they get zero or ground, we have activity on the line. and you check only the RX pin for interrupt. good luck.

[TD-er]

@josef2600 Hmm I wonder what you might be using as SW serial implementation to recommend it over HW serial. In my project I have made it user configurable to pick whatever serial port you like for the console. This includes SW serial. But I clearly see transmission errors happening when using SW serial as characters get mangled in the terminal. This already happens at 115200. For some ESP32-variants (e.g. ESP32-C3) this is hardly usable as you get lots of transmission errors. With HW serial I don't see any issues at all, not even with USB-CDC on ESP32-S2 which is notorious for performing worse than the USB variants on S2/C3 (haven't tested the C6 much yet)

[seankovacs]

Not sure why it is, but I've seen more than a few people indicate the hardware serial on the ESP32s are kinda trash. Bizzare if you ask me - wish they would address whatever the root issue is.

[TD-er]

HW serial is perfectly fine IMHO. But.... I never tried it at over 460 kbps.

The USB JTAG/HWCDC on the other hand is flashing my nodes at ~ 1 Mbps, so that's very nice when you can flash your node in under 10 sec :) (1.7 Mbps effective with compression)

Anything else I have connected to the HW serial ports was not capable of running at higher speeds, so can't say anything about stability of HW serial at such speeds.

[TD-er]

@Phetdanai What are you trying to tell us here? (and your code also doesn't check for the buffer size)