search

espressif / arduino-esp32

Arduino core for the ESP32
GNU Lesser General Public License v2.1
13.18k stars 7.33k forks source link

UART rx timeout issue! #9620

Closed hitecSmartHome closed 3 months ago

hitecSmartHome commented 3 months ago

Board

ESP32-Wrover

Device Description

-

Hardware Configuration

-

Version

v2.0.14

IDE Name

PlatformIO

Operating System

Windows10

Flash frequency

80

PSRAM enabled

yes

Upload speed

115200

Description

I'm using onReceive callback for Serial1. Sometimes the packet is truncated and I got the last one or two bytes in the next packet. No matter the packet timeout it seems.

Sketch

void setup(){
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(MODE_RS485_HALF_DUPLEX);
    Serial1.setTxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxTimeout(RX_TIMEOUT);
    Serial1.onReceive([this](void) { handleRawPacket(); },true);
    Serial1.onReceiveError([this](hardwareSerial_error_t error) { handlePacketError(error); });
}

void handleRawPacket() {
    packetByteCount = 0;
    while (Serial1.available()) {
        respBuffer[packetByteCount] = Serial1.read();
        packetByteCount++;
    }
    #if MODBUS_DEBUG
    printf("[Modbus] - Got packet: ");
    for (size_t i = 0; i < packetByteCount; i++){
        printf("0x%02x ", respBuffer[i]);
    }
    printf("\n");
    vTaskDelay(DEBUG_DELAY_MS);
    #endif
    gotPacket = true;
}

void handlePacketError(hardwareSerial_error_t error){
    // Ignore the UART_BREAK_ERROR because it is expected.
    if( error == UART_BREAK_ERROR ){ return; }
    #if MODBUS_DEBUG
        printf("\n[Modbus] - Packet error code: %d\n", error);
        printf("[Modbus] - Packet error description: %s\n", uartErrorStrings[error]);
    #endif
}

Debug Message

Just one packet in two ( usually the last two bytes is trucated and comes as a second packet ).
I have tried RX_TIMEOUT from 3 to 20. Does not matter.

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

hitecSmartHome commented 3 months ago

My basic idea:


// RX_TIMEOUT / (MBUS_BAUD / 10) = millisec
// 4 / (115200 / 10) = 3.4 ms
// 5 / (115200 / 10) = 4.3 ms
// 6 / (115200 / 10) = 5.2 ms
#define RX_TIMEOUT 4 // Have tried even RX_TIMEOUT 20 / No effect
#define RESPONSE_BUFFER_SIZE 2048

#define MBUS_BAUD 115200
#define MBUS_RX 35
#define MBUS_TX 32
#define MBUS_RTS 33

void setup(){
    Serial1.begin(MBUS_BAUD, SERIAL_8N1, MBUS_RX, MBUS_TX);
    Serial1.setPins(-1, -1, -1, MBUS_RTS);
    Serial1.setMode(MODE_RS485_HALF_DUPLEX);
    Serial1.setTxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxBufferSize(RESPONSE_BUFFER_SIZE);
    Serial1.setRxTimeout(RX_TIMEOUT);
    Serial1.onReceive([this](void) {
        // In theory, we got a whole packet.
        packetByteCount = 0;
        while (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
        }
        gotPacket = true;
    },true);
    Serial1.onReceiveError([this](hardwareSerial_error_t error) { handlePacketError(error); });
}

void loop(){
    if( gotPacket ){
        processPacket();
        packetByteCount = 0;
        gotPacket = false;
    }
}

Sometimes the onReceive cb triggers before I got a whole packet. Packets are standard modbus packets. A lot of times It got cut on the last one or two ( sometimes even three ) bytes.

hitecSmartHome commented 3 months ago

We measured the byte times with an oscilloscope and they all come at the exact same time. No delay between bytes in a frame. It seems to me when the onReceive cb is called, I can't read the whole uart data. Will test it further soon.

Meanwhile I print out the byte count in the cb

Serial1.onReceive([this](void) {
    packetByteCount = 0;
    int available = Serial1.available();
    printf("GOT PACKET COUNT: %d\n",available);
    while(available--){
        respBuffer[packetByteCount] = Serial1.read();
        packetByteCount++;
    }
    gotPacket = true;
},true);

The problem is that it does not happen every time, just sometimes. Here is a more realistic approach of mine.

MODBUS Snippet

/* add a new modbus address to a given device */
boolean Modbus::writeNewAddress(uint16_t id, uint16_t serial, uint16_t newAddress){
    const byte writeBuffSize = 13;
    uint8_t buffer[writeBuffSize];

    buffer[0] = BROADCAST_ADDRESS;
    buffer[1] = WRITE_FUNC_CODE;

    buffer[2] = highByte(WRITE_NEW_ADDRESS_REG_ADDRESS);
    buffer[3] = lowByte(WRITE_NEW_ADDRESS_REG_ADDRESS);

    buffer[4] = highByte(WRITE_NEW_ADDRESS_DATA_QUANTITY);
    buffer[5] = lowByte(WRITE_NEW_ADDRESS_DATA_QUANTITY);

    buffer[6] = uint8_t(WRITE_NEW_ADDRESS_DATA_QUANTITY*2);

    buffer[7] = highByte(id);
    buffer[8] = lowByte(id);

    buffer[9] = highByte(serial);
    buffer[10] = lowByte(serial);

    buffer[11] = highByte(newAddress);
    buffer[12] = lowByte(newAddress);

    rawWrite(buffer,writeBuffSize);
    return waitResponse();
}

/* write the raw buffer to the serial */
void Modbus::rawWrite(uint8_t *data, int length) {
    int crc16 = getCRC_16(data, length);
    uint8_t buffer[length + 2];

    memcpy(buffer, data, length);
    buffer[length] = lowByte(crc16);
    buffer[length + 1] = highByte(crc16);

    lastFuncCode = buffer[1];
    Serial1.write(buffer, length + 2);
}

/* Check if the packet is valid or not */
boolean Modbus::isValidResponse() {
    boolean crcIsOk = isValidCRC();
    boolean hasError = hasErrorByte();
    boolean hasAnyData = packetByteCount > 0;
    boolean isValid = crcIsOk && hasAnyData && !hasError;
    #if MODBUS_DEBUG
      printf("[Modbus] - Is CRC valid: %s\n", crcIsOk?"true":"false");
      printf("[Modbus] - Has error byte: %s\n", hasError?"true":"false");
      printf("[Modbus] - Has any data: %s\n", hasAnyData?"true":"false");
    #endif
    return isValid;
}

/* Wait for a response */
boolean Modbus::waitResponse() {
    while (!gotPacket) { vTaskDelay(1); }
    gotPacket = false;
    return isValidResponse();
}

MODBUS usage

/* Add new addresses to the modules on the bus */
for(initialModule module : initialModules) {
  int newAddress = getNewAddress();
  if( newAddress == -1 ){continue;}
  boolean result = modbus.writeNewAddress(module.id, module.serial, newAddress);
  if(!result){ continue; }
  addFreshModule(module.id, module.serial, newAddress);
}

Main app flow

write something -> wait for a response -> process response -> write something...

SuGlider commented 3 months ago

Sometimes the onReceive cb triggers before I got a whole packet. Packets are standard modbus packets. A lot of times It got cut on the last one or two ( sometimes even three ) bytes.

ESP32 Arduino UART is based on the UART IDF implementation. The way it works follows these "rules":

There is further information in the header file at https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/HardwareSerial.h#L215-L240

Therefore, if the MODBUS frame has, for instance, 50 to 100 bytes, it is possible to set the RxFIFOFull to 110 bytes and the callback will be able to read it all, as in the code, you have presented, would do.

But if the frame has, for instance, 300 bytes, the callback will be called for the first 120 received bytes (default value) and available() will inform that there are, at that time, 120 bytes in the buffer when it is executed. The next 120 bytes (FIFO Full) will trigger the callback again, in a next time, and the remaining 60 bytes will be triggered by RX Timeout (callback). It means that for receiving 300 bytes, the callback function would be called 3 times. The callback routine would need to assemble the MODBUS packet along the time and sequencial calls.

Let's say the a MODBUS packet would timeout a received packet if no UART communication happens after 100 ms from the last received byte. This timeout can be used to check if the complete packet has been received in a sigle execution of the onReceive() callback. Another possibility could be that the MODBUS frame has an specific field to mark the end of the packet, it could also work for waiting until the complete frame is received.

This timeout can be as high as necessary (30 seconds, for instance) or as low as the double of RX Timeout value. It runs on a separated task, therefore it doesn't delay the main sketch execution (setup()/loop()), which runs on its own task.

In order to retrieve all the, for instance, 300 bytes, it would be necessary to run a code like this:

Serial1.onReceive([this](void) {
    uint32_t packetByteCount = 0;
    const unsigned long packetTimeout_ms = 100;  // set it to whatever is necessary

    unsigned long lastByteReceivedTime = millis();
    // loop will only exit on "Packet timeout"
    while(millis() < lastByteReceivedTime + packetTimeout_ms){
        if (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
            lastByteReceivedTime = millis();
        }
    }
    printf("GOT PACKET COUNT: %d\n", packetByteCount);
    gotPacket = true;
},true);

Please check/run some examples here: https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/OnReceive_Demo/OnReceive_Demo.ino https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/RxFIFOFull_Demo/RxFIFOFull_Demo.ino https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/RxTimeout_Demo/RxTimeout_Demo.ino https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/onReceiveExample/onReceiveExample.ino https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/Serial/RS485_Echo_Demo/RS485_Echo_Demo.ino

hitecSmartHome commented 3 months ago

Thank you for the detailed answer. The problem is that I got only 8 bytes at once where it fails. I have several hardwares on the bus which will send back 8 bytes each. It never reaches the 120 byte treshold.

Can it be that the fifo is not emptied when I read the serial data and the packets are left in there? I have three scenarios in my app.

  1. Master scans the bus ( writes a broadcast packet ) -> slaves are answering one by one ( 11 byte / answer )
  2. Master starts to give new addresses to the slaves -> ( 8 byte / answer )
  3. Normal communication ( answer can be a really big frame )

In the first scenario, there is no problem at all ever. I did hundreads of tests, never fails. In the second scenario, it fails a lot of times and random packets but all 8 bytes long In the third scenario, never fails.

hitecSmartHome commented 3 months ago

If the fifo full triggers the cb i need a way to reset the fifo buffer when I got a packet.

hitecSmartHome commented 3 months ago

For example now i have 7 external hardware on my bus. If each sends back a packet thats 8*7 bytes which is 56 bytes max. If i understand correctly, the default fifo buffer is 120 bytes. Even if the Serial1.read() does not reset the fifo buffer I should have no problem reading all of the packets.

I have copyed your example to my sketch. It just frozen and the gotPacket variable never triggers.

Serial1.onReceive([this](void) {
    uint32_t packetByteCount = 0;
    const unsigned long packetTimeout_ms = 200;  // set it to whatever is necessary

    unsigned long lastByteReceivedTime = millis();
    // loop will only exit on "Packet timeout"
    while(millis() < lastByteReceivedTime + packetTimeout_ms){
        if (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
            lastByteReceivedTime = millis();
        }
    }
    if(isNewAddr){
        // Write my buffer to serial to see what we got.
        printf("GOT PACKET COUNT: %d\n",packetByteCount);
        for (size_t i = 0; i < packetByteCount; i++){
            printf("0x%02x ", respBuffer[i]);
        }
        printf("\n");
    }
    gotPacket = true;
},true);
hitecSmartHome commented 3 months ago

I modified this to this

Serial1.onReceive([this](void) {
    packetByteCount = 0;
    unsigned long lastByteReceivedTime = millis();
    while(millis() < lastByteReceivedTime + PACKET_TIMEOUT_MS_TEST){
        if (Serial1.available()) {
            respBuffer[packetByteCount] = Serial1.read();
            packetByteCount++;
            lastByteReceivedTime = millis();
        }
        if( millis() - lastByteReceivedTime > 150 ){
            break;
        }
    }
    if(isNewAddr){
        printf("GOT PACKET COUNT: %d\n",packetByteCount);
        for (size_t i = 0; i < packetByteCount; i++){
            printf("0x%02x ", respBuffer[i]);
        }
        printf("\n");
    }
    gotPacket = true;
},true);

I still got dropped bytes on the bus Tried with onlyOnTimeout set to false, to true, doesn't work.

SuGlider commented 3 months ago

Could the necessary timeout for modbus bytes to arrive be higher than 200ms? It sounds like the bytes are not arriving within the "timeout" interval.

Could you please try checkin what would be the highest timeout for your modbus? Maybe longer, like a couple seconds?

TD-er commented 3 months ago

Just to comment on the timeout.... There are quite a few Modbus devices which need about 200 msec to reply. For example SenseAir S8 may sometimes take > 200 msec to reply.

So I would suggest to start experiment with higher values first to see what your device may need.

hitecSmartHome commented 3 months ago

We had an issue with our hardwares. ( we have custom hardwares on the modbus ) Indeed there was a software bug which caused the packets to cut in half. Hovewer this break was only 1.2ms, max 3.5ms. A whole packet comes throught in max 3ms. It is not a standard modbus, we made it a little bit faster.

Here is a good request and a good response on oscilloscope

good_measure_edited

Here is a good request and a bad response on oscilloscope

bad_measure_edited

This was our bad, ofc but when I changed the rxTimeout to 5 or 6 or even to 10 this gap should have been a no problem scenario.

I have tried the rxTimeout from 4 to 10

SuGlider commented 3 months ago

rxTimeout set by bool HardwareSerial::setRxTimeout(uint8_t symbols_timeout) is not measured in milliseconds. setRxTimeout(10) means 10 UART symbols. At 9600 bps, 1 symbol is about 0.8 ms. At 115200, 1 symbol is about 0.07ms. I'm calculating it by 1000ms/baudRate * 8bits.

SuGlider commented 3 months ago

based on 

#define RX_TIMEOUT 4 // Have tried even RX_TIMEOUT 20 / No effect
#define RESPONSE_BUFFER_SIZE 2048

#define MBUS_BAUD 115200

Rx Timeout is about 0.28 milliseconds...

hitecSmartHome commented 3 months ago

So what is the calculation?

Isn't RX_TIMEOUT / (MBUS_BAUD / 10) = millisec

So with 115200 baud and 10 bit it would be (4 / (115200/10)) ?

SuGlider commented 3 months ago

The calculation is (1000 / MBUS_BAUD) BITS_PER_UART_SYMBOL RX_TIMEOUT.

For 115200 | 10 bits/Symbol | RX_TIMEOUT 4 ==> (1000/115200) 10 4 = 0.35ms

SuGlider commented 3 months ago

If the desired RX Timeout is 3.5 ms, at 115200, RX_TIMEOUT shall be 40 to 45.

hitecSmartHome commented 3 months ago

Ohhhhh. If this is the case than it was totally our fault. Sorry for the ticket and thank you very much for the help!