Bus stuck after some hours

[raf59]

Hello, I'm working on an aquarium automation project with a RaspberryPi - nodejs (master) and Arduino sensors (slaves). All the sensors are communicating with the RasPI for hours without any problem. But after some hours, the bus hangs and the master receives timeouts from each sensor. When I restart the nodejs server, nothing changes. The sensors are still timed out. I ran the DEBUG mode, following the advice of Yaacov, in order to get a more verbose output and I got the logs of the moment when the bus hangs. My problem is that I don't know how to interpret this log. It looks like a buffer overflow, but I'm not sure. And whatever the issue is, I don't know the reason, and how to solve it. I'm completely stuck in my project because of that. So, I'd be very happy if someone could help me solve this problem.

Thanks in advance !!

Here are the logs :

modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 01 03 00 00 00 01 84 0a>, modbus-serial unitid: 1, modbus-serial functionCode: 3 } +11ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 01 03 02 00 f5 78 03>, modbus-serial buffer: <Buffer 01 03 02 00 f5 78 03> } +4ms modbus-serial { action: 'emit data serial rtu buffered port', modbus-serial buffer: <Buffer 01 03 02 00 f5 78 03> } +0ms modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 06 03 00 00 00 01 85 bd>, modbus-serial unitid: 6, modbus-serial functionCode: 3 } +11ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 06 03 02 00 00 0d 84>, modbus-serial buffer: <Buffer 06 03 02 00 00 0d 84> } +4ms modbus-serial { action: 'emit data serial rtu buffered port', modbus-serial buffer: <Buffer 06 03 02 00 00 0d 84> } +1ms modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 07 03 00 00 00 01 84 6c>, modbus-serial unitid: 7, modbus-serial functionCode: 3 } +10ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 07 03 02 00 00 30 44>, modbus-serial buffer: <Buffer 07 03 02 00 00 30 44> } +3ms modbus-serial { action: 'emit data serial rtu buffered port', modbus-serial buffer: <Buffer 07 03 02 00 00 30 44> } +0ms modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 02 03 00 00 00 02 c4 38>, modbus-serial unitid: 2, modbus-serial functionCode: 3 } +11ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 02 03 04 00 19 00 33 58 e1>, modbus-serial buffer: <Buffer 02 03 04 00 19 00 33 58 e1> } +3ms modbus-serial { action: 'emit data serial rtu buffered port', modbus-serial buffer: <Buffer 02 03 04 00 19 00 33 58 e1> } +0ms modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 03 03 00 00 00 01 85 e8>, modbus-serial unitid: 3, modbus-serial functionCode: 3 } +11ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 03 02 00 35 01 93>, modbus-serial buffer: <Buffer 03 03 02 00 35 01 93> } +3ms modbus-serial { action: 'emit data serial rtu buffered port', modbus-serial buffer: <Buffer 03 03 02 00 35 01 93> } +0ms modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 04 03 00 00 00 01 84 5f>, modbus-serial unitid: 4, modbus-serial functionCode: 3 } +11ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 40 92 04 83 02 d0 f0 90 53 49>, modbus-serial buffer: <Buffer 40 92 04 83 02 d0 f0 90 53 49> } +4ms modbus-serial { action: 'emit data serial rtu buffered port', modbus-serial buffer: <Buffer 04 83 02 d0 f0> } +0ms modbus-serial { action: 'send serial rtu buffered', modbus-serial data: <Buffer 05 03 00 00 00 01 85 8e>, modbus-serial unitid: 5, modbus-serial functionCode: 3 } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92> } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 83 d0 02 d0 f0>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0> } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 04 83 02 d0 f0>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0> } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 24 80 d0 02 d0 f0 a0 20 20 36 41 92>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92> } +2ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 13 d0 02 d0 f0 ff>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff> } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 04 83 02 d0 f0 08 36 41 92>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: modbus-serial <Buffer 03 03 02 00 04 83 02 d0 f0 03 03 02 00 36 41 92 03 04 83 02 d0 f0 d2>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +2ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 04 83 02 d0 f0 10 36 41 92>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 03 02 10 d0 02 d0 f0>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 03 02 00 36 e5 0d 0a 04 83 02 d0 f0 64 0a 49>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +2ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 03 02 00 46 83 02 d0 f0 03 03 02 00 36 41 92>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 d0 02 d0 f0 49>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +2ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 04 83 02 d0 f0 10 36 41 92 03 03 02 11 83 02 d0 f0>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +1ms modbus-serial { action: 'receive serial rtu buffered port', modbus-serial data: <Buffer 03 03 02 00 56 6a 0a 04 83 02 d0 f0 00 36 41 92>, modbus-serial buffer: modbus-serial <Buffer 90 53 49 03 03 02 00 b6 d0 02 d0 f0 a0 20 20 36 41 92 03 83 d0 02 d0 f0 04 83 02 d0 f0 24 80 d0 02 d0 f0 a0 20 20 36 41 92 03 13 d0 02 d0 f0 ff 04 83 ... > } +3ms

[yaacov]

HI, thanks for the issue :heart:

Can you put here the code that fails, e.g. how you connect, how you read the data, and how you try to recover once connection start failing ?

[raf59]

Thanks a lot for your very fast reply !!

Here is the code I use. The code could be cleaner. Sorry for that. I don't try to recover the bus when the connection starts failing. I don't know how to proceed. The only way I found to recover the bus is to manually reset all the sensors. I could implement an automatic hardware reset, but I think it's not the best solution, and it would require to add a reset wire from the raspberry to the sensors. I'd like to find a software solution.

let ModbusRTU = require("modbus-serial");
let client485 = new ModbusRTU();

// Modbus connection
client485.connectRTUBuffered("/dev/ttyUSB0", { baudRate: 115200, debug: true, autoOpen: false })
    .then(function() {
        client485.setTimeout(2000);
    })
    .catch(function(e) {
        console.log(e.message);
    });

let sensorsVal = {
"TE": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"LU": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"TA": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"HA": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"DL": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"PH": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"EC": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"NI": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
"CO": { Value: "-", Min:null, Max:null, Captures: 0, Errors:0, "%Errors":0 },
};
let dev = [
    {name:"TE", id:1, unit:"°C", type:"float", register:[0,1], timeoutCount:10, statut:0},
    {name:"LU", id:6, unit:"%", type:"int", register:[0,1], timeoutCount:10, statut:0},
    {name:"NI", id:7, unit:"%", type:"int", register:[0,1], timeoutCount:10, statut:0},
    {name:"TA/HA", id:2, unit:"°C/%", type:"float,int", register:[0,2], timeoutCount:10, statut:0},
    {name:"DL", id:3, unit:"cm", type:"int", register:[0,1], timeoutCount:10, statut:0},
    {name:"PH", id:4, unit:"", type:"float", register:[0,1], timeoutCount:10, statut:0},
    {name:"EC", id:5, unit:"", type:"float", register:[0,1], timeoutCount:10, statut:0},
    {name:"CO", id:8, unit:"W", type:"int", register:[0,1], timeoutCount:10, statut:0},
];

let activeDevices = ()=>dev.filter(d=>d.statut===1);

let iter = 0;

// Return all devices if iter==0  else return active devices
let devices = ()=>{
    if(iter===0 || activeDevices().length===0) {
        return dev;
    }else{
        return activeDevices();
    }
}
let currentDeviceNum = 0;
let delayAfter = 20;
let checkEvery = 19;

// Increment the number of the current device, or reset currentDeviceNum and show sensors data
function showTable() {
    if(currentDeviceNum<devices().length-1) {
        currentDeviceNum+=1;
    }else{
        currentDeviceNum=0;
        iter = (iter<checkEvery ? iter+1 : 0);
        console.table(sensorsVal);
    }
}

// Read the value of a device
function read(idDevice) {

        let currentDevice = devices()[currentDeviceNum];
        client485.setID(idDevice);

        client485.readHoldingRegisters(...currentDevice.register)
            .finally(function() {

                //console.log(devices()[currentDeviceNum].name);

            })
            .then(function(data) {
                if(!currentDevice.statut) {
                    console.log((currentDevice.name+" CONNECTED").yellow.bold);
                }
                currentDevice.statut = 1;
                currentDevice.timeoutCount = 0;

                let dat;

                let nb=1;
                if(currentDevice.name.length>2) nb=2;

                let type,name,unit;
                for(let i=0; i<nb; i++) {
                    type = nb>1 ? currentDevice.type.split('/')[i] : currentDevice.type;
                    name = nb>1 ? currentDevice.name.split('/')[i] : currentDevice.name;
                    unit = nb>1 ? currentDevice.unit.split('/')[i] : currentDevice.unit;
                    dataa = nb>1 ? data.data[i] : data.data;

                    dat = (type==="float" ? dataa/10 : dataa/1)
                    sensorsVal[name]["Value"] = dat + unit;
                    if(sensorsVal[name]["Min"]===null || dat<sensorsVal[name]["Min"]) sensorsVal[name]["Min"] = dat;
                    if(sensorsVal[name]["Max"]===null || dat>sensorsVal[name]["Max"]) sensorsVal[name]["Max"] = dat;
                    sensorsVal[name]["Captures"]++;
                    sensorsVal[name]["%Errors"] = parseInt((sensorsVal[name]["Errors"]/(sensorsVal[name]["Captures"]+sensorsVal[name]["Errors"]))*100*100)/100;
                }

                setTimeout(()=>read(devices()[currentDeviceNum].id), delayAfter);
                showTable();
            })
            .catch(function(e) {
                                // Set the device as "disconnected" after 5 timeouts
                if(currentDevice.statut) {
                    if(currentDevice.timeoutCount===5) {
                        console.log((currentDevice.name+" DISCONNECTED").yellow.bold);
                        currentDevice.statut = 0;
                    }else{
                        currentDevice.timeoutCount++;
                    }
                }

                let nb=1;
                if(currentDevice.name.length>2) nb=2;

                let name;
                                // Split the received value if needed
                for(let i=0; i<nb; i++) {
                    name = nb>1 ? currentDevice.name.split('/')[i] : currentDevice.name;
                    sensorsVal[name]["Value"] = currentDevice.statut ? e.message : "-";
                    sensorsVal[name]["Errors"]++;
                    sensorsVal[name]["%Errors"] = parseInt((sensorsVal[name]["Errors"]/(sensorsVal[name]["Captures"]+sensorsVal[name]["Errors"]))*100*100)/100;
                }

                showTable();
                if(devices().length) {
                    read(devices()[currentDeviceNum].id);
                }else{
                    retryConnect();
                }
            });

        // Stop the script if no device found
    if(iter!==0 && !activeDevices().length) {
        console.log("No device found".red.bold);
        process.exit();
    }
}

// Retry connection every 2 sec while no device is found
let retryConnect = ()=>{
    setTimeout(()=>{
        console.log("Retry Connection");
        if(devices().length) {
            read(devices()[currentDeviceNum].id);
        }else{
            retryConnect();
        }
    }, 2000)
}

// Start the first read after 2sec
setTimeout(()=>read(devices()[currentDeviceNum].id), 2000)

[yaacov]

Did you try to add setTimeout to this read ? does it make a different ?

showTable();
if(devices().length) {
    read(devices()[currentDeviceNum].id);
}else{
    retryConnect();
}

[raf59]

I have not tried it yet but now it's done. I set it to 200ms. I'll let it run all night long and tomorrow I'll know if it works for sure. 🌜 I'll keep my fingers crossed ! Thank you for your time and patience, I guess you're very busy. 😉

[raf59]

😥 Same problem. So bad. It worked for 7 hours before stopping.

I was wondering why the buffer was getting so big and looking at the rtubufferedport.js page, I understand that the buffer is reset in the _emit method wich is not triggered in my case. I added a logging of the variables to know what condition is not met, but I can't really investigate further because of my lack of knowledge about modbus.

Would you have another idea ?

[sigma-design]

From your observations it seems that the issue might be with the slaves, not the master.

To narrow down potential problem area and as a general fault finding technique, you may consider trying the following steps (observing the changes in fault appearance, i.e. whether the fault occurs sooner or later, etc.):

1. Reduce number of physically connected slaves to 1 and/or 2.
2. Replace Arduino physical slaves with different Modbus slaves (or Modbus slave simulation software).
3. Introduce an 'idle' interval (a few seconds) before next loop cycling through slaves - this is in case if slaves' software is inferior and may require some Modbus line 'silent' time for proper operation.

Hopefully the above tests might throw some light on the issue.

[raf59]

Thank you sigma !

I'll try what you advise me. I removed all the sensors except one. I'll let it run to see if the error still occurs. I'll let you know.

About the second test, I don't have Modbus slaves except mine (with arduino). Do you have a software to advise to me in order to implement a software slave ? I've never done it before. I'll have to buy another RS485 dongle because I only have one, used for the master (RasPI).

[yaacov]

I don't have Modbus slaves except mine (with arduino).

p.s. https://github.com/yaacov/ArduinoModbusSlave was updated last week [ not sure if new version is better of has less bugs :smile: ]

[yaacov]

p.p.s :-) You can take a look at the demo, it has lots of status logs that may help too: https://github.com/yaacov/node-modbus-serial/blob/master/examples/polling_RTU.js

[raf59]

Thanks yaacov ! I'll try it tonight ! For now, I'm performing the test advised by sigma. It would be easier to diagnose if the error did occur more often... 😴

[sigma-design]

There are many Modbus slave simulation PC software products available.

Personally I have used this one (it's not free, licence costs $99): https://www.modbustools.com/modbus_slave.html

You may also explore the official Modbus site that have links to Modbus related products: http://modbus.org/tech.php

[raf59]

Thanks for the links ! I ran the script for 10 hours without any problem. I received 1 million values from the sensor and the bus didn't hang. So I retried with a second sensor and I got an error after only one hour. I retry with one only sensor to be sure. If I get the same, I'll try the new library on the 2 arduino sensors and try once again.

[sigma-design]

@raf59

Good that you can see now some changes !

Another thing which you may try is to run your system at slower baud rate, e.g. 9600 or 19200 and observe the behaviour.

Also I understand that you are using (USB-to-RS485 ?) dongle, so it is another piece of equipment in your system's Modbus communication channel - from my experience such dongles from different companies can behave differently, so you may consider trying a different one.

[raf59]

You're right. I didn't think about the dongle. I wait for another kind of dongle. As soon as I'll receive it, I'll swap it. I'm now almost sure the problem doesn't happend with a single sensor. The bus is working fine for 15 hours.

For the next steps, iI will proceed as following :

1. Try with the new Arduino library
2. Try with a lower baudrate (e.g. 9600)
3. Try the "polling rtu" example on the server side, after adapting it for 2 sensors
4. Try with another USB-RS485 dongle

Considering the time between each attempt, it might be long. I will keep you informed

Thanks again for your help ! ⭐️

[raf59]

@yaacov I tried the new library but it looks not beeing working properly. To be sure, I used the new arduino library on the slave with the supplied example (full.ino), and the pollingRTU example on the NodeJS master side. Using the pollingRTU example with the old library is working fine. I only tried with the readHoldingRegister function.

Do you want I open an issue in the ArduinoModbusSlave library ?

polling_RTU.js + Old arduino library :

polling_RTU.js + New arduino library

[yaacov]

Do you want I open an issue in the ArduinoModbusSlave library ?

Sure, that will be great :exclamation: , Thanks fot testing it :+1:

[raf59]

I followed your recommendations and it looks like it's working now. The bus didn't hang for the last 24h. I am very happy and relieved that it's finally working. ❤️ 😃 👍

I defined the baudrate at 57600 and I also used the new release of the arduino library (ArduinoModbusSlave). So, for now, I can't know if the solution comes from the library or the baudrate. I'll perform a last test with 115200 bauds to get the answer. I will report back soon.