Hardware serial sometimes doesn't send every data.

zekageri commented 2 years ago

Board

esp32-wrover-e

Device Description

Custom board, using ESP32-Wrover-E ( 16mb flash 8mb external ram )

Hardware Configuration

Ethernet -> ETH_LAN_8720 chip

RTC Module -> DS3231 chip ( i2c )

Transistor for restarting the device -> 2n2222 or similar. ( GPIO 12 )

Two hardware serial UART:

Debug Serial 0 : RX: GPIO 3 | TX: GPIO 1
Custom Modbus Serial 1 : RX: GPIO 34 | TX: GPIO 15

Version

latest master (checkout manually)

IDE Name

Platform IO

Operating System

Windows 10

Flash frequency

80Mhz

PSRAM enabled

yes

Upload speed

115200

Description

Serial1 sometimes does not send out all the bytes and i got a frame timeout in my communication because the modules that my esp communicates doesnt understand the broken message. Happens mostly on web page load or file upload via HTTP.

Communication runs on an available core in a separate task with 115200 baud rate, tries to write as soon as possible.

Sketch

#define MBUS_BAUD 115200
#define MBUS_RX 34  //   6.  PIN
#define MBUS_TX 15  //   23. PIN

void mBusSetup(){
    pinSetup();
    Serial1.begin(MBUS_BAUD,SERIAL_8N1,MBUS_RX,MBUS_TX);
}

void mSystem::pinSetup(){
    pinMode(MBUS_TX_EN, OUTPUT);
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::startWrite(){
    mBusLock();
    //Serial1.flush();
    digitalWrite(MBUS_TX_EN, HIGH);
}

void mSystem::endWrite(){
    Serial1.flush();          // does not help
    //delayMicroseconds(200); // Does not help. Dont want magic delays
    digitalWrite(MBUS_TX_EN, LOW); // TX_ENABLE pin pulled low too soon i assume.
    mBusUnLock();
}

void mSystem::rawWrite(uint8_t * data, int length){
    int crc16 = getCRC_16(data, length);
    startWrite();
    Serial1.write( data, length );
    Serial1.write( lowByte(crc16) );
    Serial1.write( highByte(crc16) );
    endWrite();
}

#define MBUS_DISCOVER_LISTEN_TIMEOUT 150 // Modules will respond back in maximum of 15ms.

void writeReadDevice(int index){
    // This function handles which device address to write and assmebles the writable data and calls
    // rawWrite. If rawWrite returns, this function waits for the response data.
    uint8_t exampleData[] = {0xFF, 0x03, 0x00, 0x06, 0x00, 0x03};
    uint8_t responseData[20]; // Maximum response data is 10-13.
    rawWrite(exampleData, sizeof(exampleData));

    // wait for the response data. The modules will respond back if the data is understandable.
    long startMs = millis();
    while( !Serial1.available() ){
        vTaskDelay(1);
        if( millis() - startMs >= MBUS_DISCOVER_LISTEN_TIMEOUT ){
            Serial.println("FRAME TIMEOUT!");
            return;
        }
    }

    int respDataCount = 0;
    while( Serial1.available() > 0 ){
        responseData[respDataCount] = Serial1.read();
        respDataCount++;
    }
}

void commLoop(){
    writeReadDevice(deviceIndex);
    deviceIndex++; // Used for communicating with different modules
    if( deviceIndex > deviceCounter ){deviceIndex = 0;}
}

void mBusLoopTask(void* parameter){
    mBusSetup();
    for(;;){
        commLoop();
        vTaskDelay(3); // The problem occours even faster if this delay is smaller.
    }
}

Debug Message

No debug message just broken data on Serial.

Other Steps to Reproduce

Try to write frequently on serial1.

I have checked existing issues, online documentation and the Troubleshooting Guide

[X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

SuGlider commented 2 years ago

@zekageri

I see you are using PlatformIO. Is your build based on Arduino Core 1.0.6? Latest PIO Arduino Core is still 1.0.6.

Would you mind testing it using Arduino IDE with latest Arduino Core (2.0.3-RC1)?

Thanks.

zekageri commented 2 years ago

This is my pio ini file

[env:arduino-esp32]
platform    = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
board       = esp32dev
framework   = arduino

platform_packages =
   framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32#master

board_build.f_cpu   = 240000000L

upload_port     = COM8
upload_speed    = 921600
monitor_speed   = 115200
monitor_filters = esp32_exception_decoder

board_build.flash_size = 16MB
board_build.flash_mode = dio
board_build.partitions = ./hsh_Partition.csv
board_build.f_flash    = 80000000L

build_flags =   -DBOARD_HAS_PSRAM
                -mfix-esp32-psram-cache-issue
                -DCORE_DEBUG_LEVEL=0
                ;-D CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY
                ;-D CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY

extra_scripts = mklittlefs/replace_fs.py

zekageri commented 2 years ago

I downloaded the new Arduino IDE 2.0.0 and i can only download the 1.0.6 espressif framework.

VojtechBartoska commented 2 years ago

Please check this guide: https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html

For using v2.0.3-rc1, you need to change your link in Preferences to Development release link.

zekageri commented 2 years ago

I know that but with PIO i'm already using the latest with this ini file:

platform    = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
board       = esp32dev
framework   = arduino
platform_packages =
   framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32#master

SuGlider commented 2 years ago

@zekageri - please check these links regarding PIO setup: https://github.com/espressif/arduino-esp32/issues/6647#issuecomment-1111792625 https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html#installing-using-platformio

zekageri commented 2 years ago

Thank you, i will try this!

zekageri commented 2 years ago

Wow @SuGlider i got a bunch of warning messages ( like really much ) and then a linker fail

c:/users/pc/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.exe: final link failed: No such file or directory
collect2.exe: error: ld returned 1 exit status
*** [.pio\build\arduino-esp32\firmware.elf] Error 1

I will try to delete platform and packages folder from .platformio and recompile

zekageri commented 2 years ago

Okay, deleted cache, packages and platform folder from .platformio inside users. The sketch compiled, i did upload it to my esp32 but the problem presist. Serial.flush() does not wait for all bytes

SuGlider commented 2 years ago

I'm pretty sure that Serial.flush() only returns when all bytes are sent. https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/esp32-hal-uart.c#L319-L332

There was an issue related to MODBUS and it was fixed there. Fixed by #6133

Are you sure you are building with the latest Arduino Core version?

zekageri commented 2 years ago

Yeah, its 4.1.0

zekageri commented 2 years ago

This is really interesting. Is there something wrong with my code? I configured Serial2 now instead of the Serial1 and commented out the microsec delay and i rarely got frame timeout now.

If there is file system access or other mid heavy process on other tasks, i got one or two frame timeouts.

zekageri commented 2 years ago

void mBusSetup(){
    pinSetup();
    Serial2.begin(MBUS_BAUD,SERIAL_8N1,MBUS_RX,MBUS_TX);
}

void mSystem::pinSetup(){
    pinMode(MBUS_TX_EN, OUTPUT);
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::startWrite(){
    digitalWrite(MBUS_TX_EN, HIGH);
}

void mSystem::endWrite(){
    Serial2.flush();
    //delayMicroseconds(400);
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::rawWrite(uint8_t * data, int length){
    int crc16 = getCRC_16(data, length);
    startWrite();
    Serial2.write( data, length );
    Serial2.write( lowByte(crc16) );
    Serial2.write( highByte(crc16) );
    endWrite();
}

#define MBUS_LISTEN_TIMEOUT 150 // Modules will respond back in maximum of 15ms.

void writeReadDevice(int index){
    // This function handles which device address to write and assmebles the writable data and calls
    // rawWrite. If rawWrite returns, this function waits for the response data.
    uint8_t exampleData[] = {0xFF, 0x03, 0x00, 0x06, 0x00, 0x03};
    uint8_t responseData[20]; // Maximum response data is 10-13.
    rawWrite(exampleData, sizeof(exampleData));

    // wait for the response data. 
    // The modules will respond back if the data is understandable.

    long startMs = millis();
    while( !Serial1.available() ){
        vTaskDelay(1);
        if( millis() - startMs >= MBUS_LISTEN_TIMEOUT ){
            Serial.println("FRAME TIMEOUT!");
            return;
        }
    }

    int respDataCount = 0;
    while( Serial1.available() > 0 ){
        responseData[respDataCount] = Serial1.read();
        respDataCount++;
    }
}

vshymanskyy commented 2 years ago

Are you absolutely sure that the output data is broken? I currently have issues reading the data with v2.0.0+, including v2.0.3-rc1. My issue is that incoming data up to 120 bytes (UART_FIFO_LEN - margin?) gets buffered (Serial1.available() returns 0), but then everything arrives in a burst.

SuGlider commented 2 years ago

There is a new API for UART called onReceive(callbackFunc)

It works as a sort of ISR... When data arrives and there is short interval of time (as long as about 11 symbols in the current baudrate), it will call right away the callback function.

It may help...

vshymanskyy commented 2 years ago

@SuGlider thank you. onReceive works for the default Serial, but doesn't work on the Serial1, which is initialized this way:

//Serial1.setRxBufferSize(512); // tried this, doesn't affect anything. data is always read in chunks of 120 bytes
Serial1.begin(9600, SERIAL_8N1, 2, 15);

On 1.0.6 i don't see such issues. Could it be an ESP-IDF issue?

UPD: ok, what I experience is similar to this: https://github.com/espressif/esp-idf/issues/8369 But in my case RS485 is not involved. Ok, will get to this with the logic analyzer.

SuGlider commented 2 years ago

Actually it works for all Serial, Serial1 and Serial2.

Just use Serial1.onReceive()

vshymanskyy commented 2 years ago

Sure. But it looks like i have problems on the signal lavel. Sorry, it is probably unrelated to this issue.

SuGlider commented 2 years ago

void mBusSetup(){
    pinSetup();
    Serial2.begin(MBUS_BAUD,SERIAL_8N1,MBUS_RX,MBUS_TX);
}

void mSystem::pinSetup(){
    pinMode(MBUS_TX_EN, OUTPUT);
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::startWrite(){
    digitalWrite(MBUS_TX_EN, HIGH);
}

void mSystem::endWrite(){
    Serial2.flush();
    //delayMicroseconds(400);
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::rawWrite(uint8_t * data, int length){
    int crc16 = getCRC_16(data, length);
    startWrite();
    Serial2.write( data, length );
    Serial2.write( lowByte(crc16) );
    Serial2.write( highByte(crc16) );
    endWrite();
}

#define MBUS_LISTEN_TIMEOUT 150 // Modules will respond back in maximum of 15ms.

void writeReadDevice(int index){
    // This function handles which device address to write and assmebles the writable data and calls
    // rawWrite. If rawWrite returns, this function waits for the response data.
    uint8_t exampleData[] = {0xFF, 0x03, 0x00, 0x06, 0x00, 0x03};
    uint8_t responseData[20]; // Maximum response data is 10-13.
    rawWrite(exampleData, sizeof(exampleData));

    // wait for the response data. 
    // The modules will respond back if the data is understandable.

    long startMs = millis();
    while( !Serial1.available() ){
        vTaskDelay(1);
        if( millis() - startMs >= MBUS_LISTEN_TIMEOUT ){
            Serial.println("FRAME TIMEOUT!");
            return;
        }
    }

    int respDataCount = 0;
    while( Serial1.available() > 0 ){
        responseData[respDataCount] = Serial1.read();
        respDataCount++;
    }
}

It seems OK, if you are sending data through Serial2 and receiving through Serial1.

SuGlider commented 2 years ago

Okay, deleted cache, packages and platform folder from .platformio inside users. The sketch compiled, i did upload it to my esp32 but the problem presist. Serial.flush() does not wait for all bytes

I just tested Serial.flush(). It only returns when FIFO is empty and all data has been sent out. It is very easy to see it when baudrate is 300bps and we send a string with about 200 characters, associating turning on/off builtin LED before/after Serial.print().

zekageri commented 2 years ago

Can it be high baud rate problem? ( 115200 ) Or can it be that other tasks interfere, and the pin is not going to low in time?

zekageri commented 2 years ago

void mSystem::startWrite(){
    digitalWrite(MBUS_TX_EN, HIGH);
}

void mSystem::endWrite(){
    Serial1.flush();
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::rawWrite(uint8_t * data, int length){
    int crc16 = getCRC_16(data, length);
    startWrite();
    Serial1.write( data, length );
    Serial1.write( lowByte(crc16) );
    // Other tasks interrupts this task in here maybe??
    Serial1.write( highByte(crc16) );
    // OR here?
    endWrite();
}

SuGlider commented 2 years ago

@zekageri - Just to make sure it is not bad configuration of the development environment, could you please update your PlatformIO version to the latest as described in https://piolabs.com/blog/news/platformio-oss-april-2022-updates.html

Thanks!

zekageri commented 2 years ago

Thank you for your response. I did update my environmnet. Currently using espressif 4.2.0 and PlatformIO 5.2.5.

This results in a strange boot loop when using default_16MB.csv with esp-wrover-kit board. The wrover kit's external ram and it's 16mb flash is absolutly necessary for me. I can't go forward if this isnt solved.

I opened a new issue for this : Boot loop issue with 16mb csv

zekageri commented 2 years ago

Boot loop problem solved. Upgraded to latest framework, issue still presist.

Frame timeouts are inconsistent. Can not lower the baudrate because the modules that i communicate with are tied to 115200.

void mSystem::startWrite(){
    //mBusLock();
    //Serial2.flush();
    digitalWrite(MBUS_TX_EN, HIGH);
}

void mSystem::endWrite(){
    Serial2.flush();
    //delayMicroseconds(200);
    digitalWrite(MBUS_TX_EN, LOW);
    //mBusUnLock();
}

void mSystem::rawWrite(uint8_t * data, int length){
    int crc16 = getCRC_16(data, length);
    startWrite();
    Serial2.write( data, length );
    Serial2.write( lowByte(crc16) );
    Serial2.write( highByte(crc16) );
    endWrite();
}

rawWrite(writeBuffer,writeBufferSize);

long startMS = millis();
while ( !Serial2.available() ){
    vTaskDelay(1);
    if( millis() - startMS > READ_WRITE_TIMEOUT ){
        Serial.printf("FRAME TIMEOUT at millis: %d\n",millis());
        return false;
    }
}

Serial:

FRAME TIMEOUT at millis: 71669
FRAME TIMEOUT at millis: 127851

It is rarely happening with the new espressif 4.2.0. But it is there.

( mostly on page refresh, or long file operations with LittleFs )

zekageri commented 2 years ago

Sometimes i get a bunch of timeouts on page close or refresh like this:

FRAME TIMEOUT at millis: 39941
FRAME TIMEOUT at millis: 40124
FRAME TIMEOUT at millis: 40304
FRAME TIMEOUT at millis: 57933
FRAME TIMEOUT at millis: 58296
FRAME TIMEOUT at millis: 58484
FRAME TIMEOUT at millis: 58675
FRAME TIMEOUT at millis: 77164
FRAME TIMEOUT at millis: 85935
FRAME TIMEOUT at millis: 86111
FRAME TIMEOUT at millis: 86289
FRAME TIMEOUT at millis: 86468
FRAME TIMEOUT at millis: 86642
FRAME TIMEOUT at millis: 89668
FRAME TIMEOUT at millis: 89844
FRAME TIMEOUT at millis: 90020
FRAME TIMEOUT at millis: 90196
FRAME TIMEOUT at millis: 90459

SuGlider commented 2 years ago

Questions:

How much is READ_WRITE_TIMEOUT ? 150ms? Why using vTaskDelay(1); ? It will delay for a whole RTOS Tick... I'm not sure but I think that it is about 100ms... How to make sure that this timeout is not due to other devices or the bus itself?

zekageri commented 2 years ago

READ_WRITE_TIMEOUT is 150ms. ( it should be max 20-30 ms since a device takes about 10-15 ms to respond )
vTaskDelay(1) is used to allow other tasks to run while that task waits for incoming bytes. ( it does not change anything if i comment it out )
We measured it with Oscilloscope. The devices on the bus never missed a single packet in extensive texting. They are not doing any other tasks just sending back their registers if the esp asks them to.

SuGlider commented 2 years ago

@vshymanskyy

My issue is that incoming data up to 120 bytes (UART_FIFO_LEN - margin?) gets buffered (Serial1.available() returns 0), but then everything arrives in a burst.

It seems that IDF has actually a special behavior when RX FIFO reaches 120 bytes (full threshold). I'm investigating it and there is a setup for "timeout always on even on FIFO Full" that is disabled by default in IDF driver. Further information in https://github.com/espressif/arduino-esp32/issues/6689#issuecomment-1146626896

SuGlider commented 2 years ago

READ_WRITE_TIMEOUT is 150ms. ( it should be max 20-30 ms since a device takes about 10-15 ms to respond )

vTaskDelay(1) is used to allow other tasks to run while that task waits for incoming bytes. ( it does not change anything if i comment it out )

We measured it with Oscilloscope. The devices on the bus never missed a single packet in extensive texting. They are not doing any other tasks just sending back their registers if the esp asks them to.

Possible relationship with https://github.com/espressif/arduino-esp32/issues/6689#issuecomment-1146626896

zekageri commented 2 years ago

Meanwhile i throttled my communication task and i found out that if i write on serial every 4.7ms the esp sends out all the packets and the modules on the bus are happily answer every time. This delay is fluid because i did not used timers, just the good old millis(). But it definitely helped. Ofc it is a workaround. I want my communication task to be as fast as possible since there can be dozens of modules on the bus and it can take a while to write all of them ( and wait for every response )

zekageri commented 2 years ago

The processors inside the modules are PICKs, they are not the fastest, so if the esp wants like 20 continous register values, that can delay the whole communication. And there are GPIO-s on some of the modules too. So it would be really good to not include this 4700 micro delay.

SuGlider commented 2 years ago

@zekageri

Regarding the time it take to return from Serial2.available(), please try with this code (change it to Serial2 if necessary):

void mBusSetup(){
    pinSetup();
    Serial1.begin(MBUS_BAUD,SERIAL_8N1,MBUS_RX,MBUS_TX);
    Serial1.setRxTimeout(1);

    // Not sure if the code lline below improves response time for Serial1.available() 
    // please make a test with it and another without it
    uart_set_always_rx_timeout(1, true); // 1 is for Serial1, in this case
}

zekageri commented 2 years ago

Thank you for the suggesrion. i will try this now.

zekageri commented 2 years ago

identifier "uart_set_always_rx_timeout" is undefined

Do i have to include something for this to work?

error: 'uart_set_always_rx_timeout' was not declared in this scope    
     uart_set_always_rx_timeout(2, true);

zekageri commented 2 years ago

With this in setup:

Serial2.begin(MBUS_BAUD,SERIAL_8N1,MBUS_RX,MBUS_TX);
Serial2.setRxTimeout(1);
//uart_set_always_rx_timeout(1, true);

and commented out delay in loop

//if( micros() - lastWriteMicros >= MODBUS_MIN_WRITE_TIME_MICRO ){
    //lastWriteMicros = micros();
    writeReadDevice(deviceIndex);
    deviceIndex++;
    if( deviceIndex > deviceCounter ){ deviceIndex = 0; vTaskDelay(1); }
//}

it is much worse.

Sample from my debug output:

FRAME TIMEOUT at millis: 144274 module id: 20
FRAME TIMEOUT at millis: 144334 module id: 179
FRAME TIMEOUT at millis: 144430 module id: 20
FRAME TIMEOUT at millis: 144490 module id: 179
FRAME TIMEOUT at millis: 144559 module id: 177
FRAME TIMEOUT at millis: 144637 module id: 20
FRAME TIMEOUT at millis: 144697 module id: 179
FRAME TIMEOUT at millis: 144766 module id: 177
FRAME TIMEOUT at millis: 144846 module id: 20
FRAME TIMEOUT at millis: 144906 module id: 179
FRAME TIMEOUT at millis: 144975 module id: 177
FRAME TIMEOUT at millis: 145053 module id: 20
FRAME TIMEOUT at millis: 145119 module id: 177
FRAME TIMEOUT at millis: 145183 module id: 177
FRAME TIMEOUT at millis: 145262 module id: 20
FRAME TIMEOUT at millis: 145328 module id: 177
FRAME TIMEOUT at millis: 145392 module id: 177
FRAME TIMEOUT at millis: 145499 module id: 177
FRAME TIMEOUT at millis: 145590 module id: 179
FRAME TIMEOUT at millis: 145687 module id: 20
FRAME TIMEOUT at millis: 145747 module id: 179
FRAME TIMEOUT at millis: 145816 module id: 177
FRAME TIMEOUT at millis: 145895 module id: 20
FRAME TIMEOUT at millis: 145966 module id: 179

In every 200 ms or so, there is still a timeout on a random module.

zekageri commented 2 years ago

If i use Serial2.setRxTimeout(1); i got frame timeouts even with the 4700 microsec delay.

zekageri commented 2 years ago

So my task looks like this:

void mBusLoopTask(void* parameter){
    hsh_modbus.setup();
    hsh_modbus.ready = true;
    for(;;){
         hsh_modbus.loop();
    }
}

Loop looks like this:

void mSystem::loop(){
    scanBus(); // <-- waits for a flag
    reinitSavedDevices(); // <-- waits for a flag
    if( !shouldScanBus && !reinitingSavedDevices && !hsh_Server.firmwareUpdating && !hsh_Performance.isHwResetRunning() ){
        // MODBUS_MIN_WRITE_TIME_MICRO  -> 4700
        if( micros() - lastWriteMicros >= MODBUS_MIN_WRITE_TIME_MICRO ){
            lastWriteMicros = micros();
            writeReadDevice(deviceIndex); // <-- actual write function you see above. Assemble, write and wait for the packet back.
            deviceIndex++;
            if( deviceIndex > deviceCounter ){ deviceIndex = 0; vTaskDelay(1); }
        }

        hsh_modules.sendPeriodicModuleInfo(); // <-- every couple secs with millis() ( non blocking )
        faultReset();                            // <-- waits for a queue
        consumeMaxAmper();           // <-- waits for a queue
        consumeThermCalibration(); // <-- waits for a queue
    }else{
        vTaskDelay(100);
    }
}

Write function important part:

rawWrite(writeBuffer,writeBufferSize);
long startMS = millis();
while ( !Serial2.available() ){
    //vTaskDelay(1); // Not used. Changes nothing.

    // READ_WRITE_TIMEOUT -> 55ms ( for now )
    if( millis() - startMS > READ_WRITE_TIMEOUT ){
        Serial.printf("FRAME TIMEOUT at millis: %d module id: %d\n",millis(),modules[index]->id);
        return false;
    }

}
while ( Serial2.available() ){
    respBuffer[respDataCount] = Serial2.read();
    respDataCount++;
}

Serial1 or Serial2 does not change anything.

SuGlider commented 2 years ago

@zekageri - I found out that some power meters send BREAK (holds the line Low for more than 11 bits in the current baudrate) to UART at the end of a data frame.

It seems to cause this delay that you are reporting. Do you think that this is your case?

SuGlider commented 2 years ago

identifier "uart_set_always_rx_timeout" is undefined

Do i have to include something for this to work?
error: 'uart_set_always_rx_timeout' was not declared in this scope    
     uart_set_always_rx_timeout(2, true);

it is necessary to include "driver/uart.h" #include "driver/uart.h"

I think that both (setRxTimeout(1) and uart_set_always_rx_timeout(2, true)) may improve the timing. The idea is to force available() to always have data in the timeout of 1 symbol at the current UART baudrate. So for 9600 bauds, 1 symbol with 11 bits (parity, stop bits etc), the timeout to get data available would be about 1ms.

zekageri commented 2 years ago

Thank you for the tremendous help. The modules does not send BREAK, i'm sure of it. They are configured for 8 bit and a stop bit. I can monitor their response with multiway v10 which is a software to test modbus communcation for PC. If the esp is out of the bus and i send commands to the modules manually with multiway, i can see that they answering correctly every time. The Multiway software is configured for the same settings which is 115200 baud, no parity, 8 data bit and 1 stop bit.

I included "driver/uart.h", commented out the delay again and on setup i set both

Serial2.setRxTimeout(1);
uart_set_always_rx_timeout(2, true);

The frame timeout is here again.

But i noticed something. Regardless of these two lines, my task loop looks like this:

void mBusLoopTask(void* parameter){
    hsh_modbus.setup();
    hsh_modbus.ready = true;
    for(;;){
         //if( micros() - lastWriteMicros >= MODBUS_MIN_WRITE_TIME_MICRO ){
            //lastWriteMicros = micros();
            writeReadDevice(deviceIndex); // <-- actual write function you see above. Assemble, write and wait for the packet back.
            deviceIndex++;
            if( deviceIndex > deviceCounter ){ deviceIndex = 0; vTaskDelay(1); }
       // }
    }
}

Notice the vTaskDelay(1); for task switching happens if it writed every module. So i wanted the task switch to happen only if the esp writed every module. This results in a lot of FRAME_TIMEOUT ( usually in every 100ms or so ).

If i modify my task loop to this:

void mBusLoopTask(void* parameter){
    hsh_modbus.setup();
    hsh_modbus.ready = true;
    for(;;){
         //if( micros() - lastWriteMicros >= MODBUS_MIN_WRITE_TIME_MICRO ){
            //lastWriteMicros = micros();
            vTaskDelay(1);
            writeReadDevice(deviceIndex); // <-- actual write function you see above. Assemble, write and wait for the packet back.
            deviceIndex++;
            if( deviceIndex > deviceCounter ){ deviceIndex = 0; }
       // }
    }
}

The FRAME_TIMEOUTS happens only a couple of seconds on a random module, but now i have CRC Mismatch too.

Sample from the debug output:

FRAME TIMEOUT at millis: 3431308 module id: 20
new write CRC Mismatch. Got: FC00 Calculated: E140
new write CRC Mismatch. Got: FC00 Calculated: 21B1
new write CRC Mismatch. Got: FC00 Calculated: E140
IO - Disconnected!
new write CRC Mismatch. Got: FC00 Calculated: 21B1
new write CRC Mismatch. Got: FC00 Calculated: 21B1
IO - Disconnected!
FRAME TIMEOUT at millis: 3435624 module id: 20
IO - Disconnected!
new write CRC Mismatch. Got: FD00 Calculated: ED80
FRAME TIMEOUT at millis: 3438019 module id: 30
new write CRC Mismatch. Got: FD00 Calculated: ED80
new write CRC Mismatch. Got: FC00 Calculated: E140
IO - Disconnected!
IO - Disconnected!
new write CRC Mismatch. Got: FC00 Calculated: E140
new write CRC Mismatch. Got: FC00 Calculated: 21B1

So if i let the task switch frequently ( after every write ) the response data is broken a lot of the times, and sometimes it is even timeouts.

zekageri commented 2 years ago

If i let TickType_t 2 delay for the task between every write, no CRC mismatch and no frame timeout. ( only if i load a heavy webpage )

Like this:

void mBusLoopTask(void* parameter){
    hsh_modbus.setup();
    hsh_modbus.ready = true;
    for(;;){
        vTaskDelay(2);
        writeReadDevice(deviceIndex); // <-- actual write function you see above. Assemble, write and wait for the packet back.
        deviceIndex++;
        if( deviceIndex > deviceCounter ){ deviceIndex = 0; }
    }
}

But ofc this results in a somewhat slow communication, since i'm monitoring push buttons and their times too which are wired to the modules. I can live with that, there are ways to eliminate these times with software adjustment but for perfect timing, i would prefer faster writes. :/

There are 8 modules on the bus right now. This results in a 8x2 tick delay for a module. This vTaskDelay() can be any number, since i don't know how much time it takes to return to this task. It could be 15 or 20 or more ms in my understanding. So if i count with 15ms between writes the last module response would be 8x15ms after the communication starts. So it is 120ms between two write for a module. If the user increases the number of modules ( there can be any number of modules on the bus since the ID's are hand out automatically and stored in the fs ) let's say the user has 20 modules on the bus, it can take 300ms for one module between writes.

But this is an ideal time since i don't know how much the RTOS works with other tasks. But this 300ms delay can affect a lot of things.

SuGlider commented 2 years ago

Just to get us in the same page:

FreeRTOS runs the tasks at 1000Hz by default in Arduino. https://github.com/espressif/arduino-esp32/blob/master/tools/sdk/esp32/sdkconfig#L943 vTaskDelay(1) means delaying at least 1ms, depending on how the processing goes.

But, this time depends on the other Tasks that are running in the ESP32. I think that your project has ETH as well... The ISRs and other routines with higher priority may interfere with this timing. It is hard to make Arduino work in a very deterministic way with several peripherals, tasks and so on.

About the MODBUS environment:

How exactly is this wired? Are you using a RS-485 module? Which one? Does it use ESP32 RTS pin in order to control DE/RE pin of the RS485 module? Or DE/RE (if any) are in shortcut and controlled by MBUS_TX_EN pin?

zekageri commented 2 years ago

Yes, i aggree it can be hard to tell with so many peripherials. Yes, i use ETH and a lot of other things, and i have a lot of other tasks.

Modbus is using an RS-485 converter. I can not tell now which one it is because it itn't a complett module but an SMD IC. ( i will look it up ) The converter uses the esp MBUS_TX_EN pin to determine if it has to switch to rx or tx mode, which will be pulled low by my software when it sent out all the packets and high when it starts sending. I don't use any built in pin for this. Should i?

And again, i can not use the built in modbus library because the protocoll slighly modified in my environment and some things can not work. It isn't a standard modbus protocoll. But the timing and the packets are the same.

zekageri commented 2 years ago

As you can see in my examples before, i have these functions to write to the serial:

// Pull the converter pin to HIGH to indicate the start of the packets.
void mSystem::startWrite(){
    digitalWrite(MBUS_TX_EN, HIGH);
}

// Pull the converter pin to LOW to indicate the end of the packets and flush Serial to send all the data that is left in the buffer.
void mSystem::endWrite(){
    Serial1.flush();
    digitalWrite(MBUS_TX_EN, LOW);
}

void mSystem::rawWrite(uint8_t * data, int length){
    // Compute the crc for the data.
    int crc16 = getCRC_16(data, length);

    startWrite();

    // Add the whole data to Serial buffer.
    Serial1.write( data, length );

    // Add CRC low and high byte to the Serial buffer.
    Serial1.write( lowByte(crc16) );
    Serial1.write( highByte(crc16) );

    endWrite();
}

I did not specify any pin in Serial1.begin()

SuGlider commented 2 years ago

These fucntions should work fine... What exactly is the issue you are seeing with it?

I tested it here for about 12 horus sending data, using ESP32 with DIO (QIO doesn't work with default pins 9 and 10 for Serial1 beacuse these pins are used for accessing the flash). It works fine and fast, no delay, no data loss.

zekageri commented 2 years ago

Sorry i meant i did not specify any data controll pin which would internally go high on write and low on flush. I do it myself. My pins are the following for serial

#define MBUS_RX 34 // 6. PIN #define MBUS_TX 15 // 23. PIN #define MBUS_TX_EN 5 #define MODBUS_MIN_WRITE_TIM.E_MICRO 5000

The issue iam seeing is byte loss when writing with 115200 baudrate continously

SuGlider commented 2 years ago

@zekageri

I would like to propose a fix for Serial2.read() and Serial2.available() that I think would improve UART response time:

// Necessary include for testing the fix
#include "driver/uart.h"

void setup() {
  // for example, start Serial2 - UART2
  Serial2.begin(115200);

  // right after starting UART2, add this code:
  uart_intr_config_t uart_intr = {
      .intr_enable_mask = (0x1<<0) | (0x8<<0),  // UART_INTR_RXFIFO_FULL | UART_INTR_RXFIFO_TOUT,
      .rx_timeout_thresh = 1,
      .txfifo_empty_intr_thresh = 10,
      .rxfifo_full_thresh = 112,
  };
  uart_intr_config((uart_port_t) 2, &uart_intr);  // Two is the UART number for Arduino Serial
}

Could you please test it and let me know. Thanks.

zekageri commented 2 years ago

Thank you i will try it now.

zekageri commented 2 years ago

I have got some compile errors with this:

src/utilities/modBus.cpp: In member function 'void mSystem::testSerialSetup()':
src/utilities/modBus.cpp:1506:27: error: 'UART_INTR_RXFIFO_FULL' was not declared in this scope
       .intr_enable_mask = UART_INTR_RXFIFO_FULL | UART_INTR_RXFIFO_TOUT,
                           ^~~~~~~~~~~~~~~~~~~~~
src/utilities/modBus.cpp:1506:27: note: suggested alternative: 'UART_BUFFER_FULL'
       .intr_enable_mask = UART_INTR_RXFIFO_FULL | UART_INTR_RXFIFO_TOUT,
                           ^~~~~~~~~~~~~~~~~~~~~
                           UART_BUFFER_FULL
src/utilities/modBus.cpp:1506:51: error: 'UART_INTR_RXFIFO_TOUT' was not declared in this scope
       .intr_enable_mask = UART_INTR_RXFIFO_FULL | UART_INTR_RXFIFO_TOUT,
                                                   ^~~~~~~~~~~~~~~~~~~~~
src/utilities/modBus.cpp:1506:51: note: suggested alternative: 'UART_FIFO_OVF'
       .intr_enable_mask = UART_INTR_RXFIFO_FULL | UART_INTR_RXFIFO_TOUT,
                                                   ^~~~~~~~~~~~~~~~~~~~~
                                                   UART_FIFO_OVF*** [.pio\build\esp-wrover-kit\src\utilities\modBus.cpp.o] Error 1

#include "driver/uart.h"

void mSystem::testSerialSetup(){
    uart_intr_config_t uart_intr = {
      .intr_enable_mask = UART_INTR_RXFIFO_FULL | UART_INTR_RXFIFO_TOUT,
      .rxfifo_full_thresh = 112,
      .rx_timeout_thresh = 1,
      .txfifo_empty_intr_thresh = 10,
    };
    uart_intr_config((uart_port_t) 1, &uart_intr);
}

void mSystem::setup(){
    Serial1.begin(MBUS_BAUD,SERIAL_8N1,MBUS_RX,MBUS_TX);
    testSerialSetup();
}

By the way here is all my task initialization. I try to isolate the modbus task from the rest.

void initTasks(){
    fileSys.init(1,2,FILE_SYS_STACK_SIZE);
    hsh_modbus.init(-1,4,MODBUS_SYS_STACK_SIZE);
    networkSys.init(1,2,NETWORK_SYS_STACK_SIZE);
    hshDisplay.init(1,2,DISPLAY_SYS_STACK_SIZE);
    hsh_Server.init(1,2,SERVER_SYS_STACK_SIZE);
    hsh_Performance.init(1,2,PERFORMANCE_SYS_STACK_SIZE);
    hsh_GeoSystem.init(1,2,GEO_SYS_STACK_SIZE);
    hsh_timeSystem.init(1,2,TIME_SYS_STACK_SIZE);
    zones.init(1,3,ZONES_SYS_STACK_SIZE);
}

espressif / arduino-esp32