Truncated UART packets when accessing flash or using the `httpd` task

[hitecSmartHome]

Board

esp32 wrover

Device Description

psram, eth, flash etc..

Hardware Configuration

psram, eth, flash etc..

Version

latest master (checkout manually)

IDE Name

PlatformIO

Operating System

Windows 10

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

I often get truncated packets on uart when accessing either the flash with LittleFS wrapper or when making an http call from client to esp32. The way I test this is that I get an interrupt from Serial1.onReceive() with 0x00 data in it when I save something to flash or when making an http request. http server uses the IDF server implementation as well as the arduino Serial wrapper.

I expect the onReceive function to call my cb only when the set timeout is triggered. It is UART_SYMBOL_TIMEOUT 1 in this case. The problem is triggered every time when I write to flash. I can access menuconfig so I made sure that the UART functions and variables are in IRAM and interrupts too.

Sketch

void Modbus::init() {
    Serial1.setRxBufferSize(MAX_RX_BUFFER_SIZE);
    Serial1.setTxBufferSize(MAX_MBUS_DATA_LENGTH);
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(UART_MODE_RS485_HALF_DUPLEX);
    Serial1.setRxTimeout(MBUS_RX_TIMEOUT);

    Serial1.onReceive(
        std::bind(&Modbus::handlePacket, this),
        PACKET_TRIGGER_ONLY_ON_TIMEOUT
    );
    Serial1.onReceiveError(
        std::bind(&Modbus::handleReceiveError, this, std::placeholders::_1)
    );
}

Debug Message

Debug messages coming from my API

E (39603) Modbus: CRC error in response packet: 0x00
E (39603) Modbus: Invalid packet. Can't process it.
Raw Packet: 00
E (39608) HardwareHandler: Packet error code: 1

Other Steps to Reproduce

Give repeated modbus packets to your esp, wait for them and access the flash meanwhile.

I have checked existing issues, online documentation and the Troubleshooting Guide

• [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[SuGlider]

onReceive() will be called only when: 1- The UART FIFO is full. 2- When RX timeout occurs.

From the comments in https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/HardwareSerial.h

  // onReceive will setup a callback that will be called whenever an UART interruption occurs (UART_INTR_RXFIFO_FULL or UART_INTR_RXFIFO_TOUT)
  // UART_INTR_RXFIFO_FULL interrupt triggers at UART_FULL_THRESH_DEFAULT bytes received (defined as 120 bytes by default in IDF)
  // UART_INTR_RXFIFO_TOUT interrupt triggers at UART_TOUT_THRESH_DEFAULT symbols passed without any reception (defined as 10 symbols by default in IDF)
  // onlyOnTimeout parameter will define how onReceive will behave:
  // Default: true -- The callback will only be called when RX Timeout happens.
  //                  Whole stream of bytes will be ready for being read on the callback function at once.
  //                  This option may lead to Rx Overflow depending on the Rx Buffer Size and number of bytes received in the streaming
  //         false -- The callback will be called when FIFO reaches 120 bytes and also on RX Timeout.
  //                  The stream of incommig bytes will be "split" into blocks of 120 bytes on each callback.
  //                  This option avoid any sort of Rx Overflow, but leaves the UART packet reassembling work to the Application.
  void onReceive(OnReceiveCb function, bool onlyOnTimeout = false);

What is the value of PACKET_TRIGGER_ONLY_ON_TIMEOUT?

[SuGlider]

Could it be a memory allocation issue? Please provide a minimum sketch that can be used to confirm the issue and to debug it.

[hitecSmartHome]

PACKET_TRIGGER_ONLY_ON_TIMEOUT is true in this case. The fifo can not be full since I got messages only when asked and no message is bigger than the fifo

[hitecSmartHome]

These are standard modbus messages

[SuGlider]

OK. I can't relate UART interrupts to flash accessing / httpd task. From your description, it sounds like this is an Arduino as IDF component project with specific IDF sdkconfig settings that may be very particular to this project.

Could you please provide a minimum sketch that can be built using Arduino Core standard sdkconfig that demonstrates such issue?

[SuGlider]

What is the value of MAX_RX_BUFFER_SIZE, MAX_MBUS_DATA_LENGTH and MBUS_BAUD?

[SuGlider]

OK. I can't relate UART interrupts to flash accessing / httpd task. From your description, it sounds like this is an Arduino as IDF component project with specific IDF sdkconfig settings that may be very particular to this project.

Could you please provide a minimum sketch that can be built using Arduino Core standard sdkconfig that demonstrates such issue?

Could it be an issue with flash access / httpd tasks? Arduino setup()/loop() will run under the lowest FreeRTOS priority. By other hand onReceive() related task will run under priority ARDUINO_SERIAL_EVENT_TASK_PRIORITY. This can be set as a global #define for the whole project.

#ifndef ARDUINO_SERIAL_EVENT_TASK_PRIORITY
#define ARDUINO_SERIAL_EVENT_TASK_PRIORITY (configMAX_PRIORITIES - 1)
#endif

You can also change the ESP32-WROVER CPU that is running the UART event Loop with ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE

#ifndef ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE
#define ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE -1
#endif

Or change the Task Stack size with ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE

#ifndef ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE
#define ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE 2048
#endif

[SuGlider]

About stack size... onReceive() callback will be called from a separated UART Event Task. If the callback function overflows the Task stack, it can cause a similar issue.

[hitecSmartHome]

#define MBUS_BAUD 115200
#define PACKET_TRIGGER_ONLY_ON_TIMEOUT true
#define MBUS_RX_TIMEOUT 1
#define MAX_MBUS_DATA_LENGTH 256
#define MAX_RX_BUFFER_SIZE 80

• I don't have any arduino setup and loop. I have disabled them in the menuconfig.
• It is not so specific build, the latest arduino core and the latest idf.

This is from my sdkconfig.

CONFIG_ARDUINO_SERIAL_EVENT_RUN_NO_AFFINITY=y
CONFIG_ARDUINO_SERIAL_EVENT_TASK_RUNNING_CORE=-1
CONFIG_ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE=4048
CONFIG_ARDUINO_SERIAL_EVENT_TASK_PRIORITY=24

I can increase the event stack size for testing and also the rx buffer size.

Could you please provide a minimum sketch that can be built using Arduino Core standard sdkconfig that demonstrates such issue?

I will create one for sure because I want to get to the bottom of this.

[hitecSmartHome]

Here is my whole sdkconfig: https://pastebin.com/w4wPRxDK

[SuGlider]

define MAX_RX_BUFFER_SIZE 80

Serial1.setRxBufferSize(MAX_RX_BUFFER_SIZE);

It can't be lower than 129. Please enable DEBUG level at least to INFO in order to check error and warning messages.

From HardwareSerial Source Code:

// minimum total RX Buffer size is the UART FIFO space (128 bytes for most SoC) + 1. IDF imposition.
size_t HardwareSerial::setRxBufferSize(size_t new_size) {

  if (_uart) {
    log_e("RX Buffer can't be resized when Serial is already running. Set it before calling begin().");
    return 0;
  }

  if (new_size <= SOC_UART_FIFO_LEN) {
    log_w("RX Buffer set to minimum value: %d.", SOC_UART_FIFO_LEN + 1);  // ESP32, S2, S3 and C3 means higher than 128
    new_size = SOC_UART_FIFO_LEN + 1;
  }

  _rxBufferSize = new_size;
  return _rxBufferSize;
}

[hitecSmartHome]

Well, I see that it changes the value back.

[hitecSmartHome]

Firstly

• Increased MAX_RX_BUFFER_SIZE to 256
• Increased CONFIG_ARDUINO_SERIAL_EVENT_TASK_STACK_SIZE to 8000
• Increased log level verbosity to DEBUG

I couldn't see shit so decreased debug back to info. I could reproduce it without flash calls.

I (117321) System: Getting task info
E (117329) Modbus: CRC error in response packet: 0x00
E (117329) Modbus: Invalid packet. Can't process it.
Raw Packet: 00
E (117334) HardwareHandler: Packet error code: 1

E (117339) Modbus: CRC error in response packet: 0x14
E (117345) Modbus: Invalid packet. Can't process it.
Raw Packet: 14 00 00 ed 85
E (117353) HardwareHandler: Packet error code: 1

The packet also has missing bytes and it is splitted. The task priority is default 19 and it is amongst the biggest priority tasks the system has.

uart_event_task is priority 24 wifi is 23 esp_timer is 22 sys_evt is 20 and arduino_events is 19

I can increase the arduino_events task further hovewer. ( I want to try these things before I go write a reproduction example because it will probably involve http requests. )

[hitecSmartHome]

For some strange reason, in the menuconfig under Arduino configuration the Serial Event task priority shows up as 24 but when I do a check with uxTaskGetSystemState(taskStatusArray, uxArraySize, &ulTotalRunTime); it reports the arduino_events priority as 19. Strange...

[hitecSmartHome]

Does Arduino Serial Event task the same as arduino_events ?

[SuGlider]

Does Arduino Serial Event task the same as arduino_events ?

No. It is a separated task and settings: https://github.com/espressif/arduino-esp32/blob/0d5d50eb4186cb5ef0b7c108f87fae1cb688b84f/cores/esp32/HardwareSerial.cpp#L14-L24

https://github.com/espressif/arduino-esp32/blob/0d5d50eb4186cb5ef0b7c108f87fae1cb688b84f/cores/esp32/HardwareSerial.cpp#L123-L132

[hitecSmartHome]

So the ARDUINO_SERIAL_EVENT_TASK name is uart_event_task. In this case, the uart_event_task priority is 24 for me...

What does arduino_events task do? Like OTA and things like that?