search

DigitalShoestringSolutions / PowerMonitoring

GNU General Public License v3.0
0 stars 1 forks source link

Modbus Error: [Input/Output] ERROR: No response received after 3 retries #43

Open tobyaharris opened 1 month ago

tobyaharris commented 1 month ago

In debugging a customer's deployment that has very poor (<50%) data collection uptime, there's a lot of this message in the logs.

What's causing this and what can be done about it?

In full (it's hard to see where one situation ends and another begins in the logs):

current-sensing-1      | INFO:main:Restarting Building block measure
current-sensing-1      | INFO:main.measure:+---Started
current-sensing-1      | INFO:main.measure:+---Loading Modules
current-sensing-1      | INFO:main.measure:Loaded: 
core.interface_modules.modbus > ModbusTCPAsync
current-sensing-1      | INFO:main.measure:Loaded: 
core.device_modules.multi_function_meter_HOBUT > HOBUT_850_LTHN
current-sensing-1      | INFO:main.measure:Loaded: 
core.calculation_modules.gen_constants > ConstantSet
current-sensing-1      | INFO:main.measure:Loaded: 
core.measurement_modules.gen_sample > SingleSample
current-sensing-1      | INFO:main.measure:+---Initialising Modules
current-sensing-1      | WARNING:pymodbus.logging:Failed to connect 
[Errno 101] Connect call failed ('192.168.0.7', 502)
current-sensing-1      | INFO:main.measure:+---Starting Loop
current-sensing-1      | ERROR:core.interface_modules.modbus:ERROR: 
exception in pymodbus Modbus Error: [Connection] Not 
connected[AsyncModbusTcpClient 192.168.0.7:502]
current-sensing-1      | 
ERROR:core.device_modules.multi_function_meter_HOBUT:Traceback (most 
recent call last):
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 87, in sample
current-sensing-1      |     readings[self.varI1] = await 
self.read_modbus_register(self.regI1)
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 135, in 
read_modbus_register
current-sensing-1      |     return await result
current-sensing-1      |   File "/app/core/interface_modules/modbus.py", 
line 61, in read_register
current-sensing-1      |     raise exc
current-sensing-1      |   File "/app/core/interface_modules/modbus.py", 
line 57, in read_register
current-sensing-1      |     modbus_response = await 
self.modbus_client.read_input_registers(address=register_addr, 
count=int(count),
current-sensing-1      |   File 
"/usr/local/lib/python3.9/site-packages/pymodbus/client/mixin.py", line 
117, in read_input_registers
current-sensing-1      |     return self.execute(
current-sensing-1      |   File 
"/usr/local/lib/python3.9/site-packages/pymodbus/client/base.py", line 
179, in execute
current-sensing-1      |     raise ConnectionException(f"Not 
connected[{self!s}]")
current-sensing-1      | pymodbus.exceptions.ConnectionException: Modbus 
Error: [Connection] Not connected[AsyncModbusTcpClient 192.168.0.7:502]
current-sensing-1      |
current-sensing-1      | ERROR:core.sensing_stack:Error during sampling: 
Traceback (most recent call last):
current-sensing-1      |   File "/app/core/sensing_stack.py", line 28, 
in execute
current-sensing-1      |     sample_dict = await sample_resp
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 128, in 
sample
current-sensing-1      |     raise e
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 87, in sample
current-sensing-1      |     readings[self.varI1] = await 
self.read_modbus_register(self.regI1)
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 135, in 
read_modbus_register
current-sensing-1      |     return await result
current-sensing-1      |   File "/app/core/interface_modules/modbus.py", 
line 61, in read_register
current-sensing-1      |     raise exc
current-sensing-1      |   File "/app/core/interface_modules/modbus.py", 
line 57, in read_register
current-sensing-1      |     modbus_response = await 
self.modbus_client.read_input_registers(address=register_addr, 
count=int(count),
current-sensing-1      |   File 
"/usr/local/lib/python3.9/site-packages/pymodbus/client/mixin.py", line 
117, in read_input_registers
current-sensing-1      |     return self.execute(
current-sensing-1      |   File 
"/usr/local/lib/python3.9/site-packages/pymodbus/client/base.py", line 
179, in execute
current-sensing-1      |     raise ConnectionException(f"Not 
connected[{self!s}]")
current-sensing-1      | pymodbus.exceptions.ConnectionException: Modbus 
Error: [Connection] Not connected[AsyncModbusTcpClient 192.168.0.7:502]
current-sensing-1      |
current-sensing-1      | ERROR:main.measure:Sample Error for device 
mfm1: Modbus Error: [Connection] Not connected[AsyncModbusTcpClient 
192.168.0.7:502] - pausing for 10 seconds
current-sensing-1      | INFO:main.mqtt_out:pub topic:error/sensing_dc 
msg:{'type': 'device', 'id': 'mfm1', 'reason': 'Modbus Error: 
[Connection] Not connected[AsyncModbusTcpClient 192.168.0.7:502]', 
'timestamp': '2024-10-06T10:11:16.418573+00:00'}
current-sensing-1      | ERROR:core.interface_modules.modbus:ERROR: 
exception in pymodbus Modbus Error: [Connection] Not 
connected[AsyncModbusTcpClient 192.168.0.7:502]
current-sensing-1      | 
ERROR:core.device_modules.multi_function_meter_HOBUT:Traceback (most 
recent call last):
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 87, in sample
current-sensing-1      |     readings[self.varI1] = await 
self.read_modbus_register(self.regI1)
current-sensing-1      |   File 
"/app/core/device_modules/multi_function_meter_HOBUT.py", line 135, in 
read_modbus_register
current-sensing-1      |     return await result
current-sensing-1      |   File "/app/core/interface_modules/modbus.py", 
line 61, in read_register
current-sensing-1      |     raise exc
current-sensing-1      |   File "/app/core/interface_modules/modbus.py", 
line 57, in read_register
current-sensing-1      |     modbus_response = await 
self.modbus_client.read_input_registers(address=register_addr, 
count=int(count),
current-sensing-1      |   File 
"/usr/local/lib/python3.9/site-packages/pymodbus/client/mixin.py", line 
117, in read_input_registers
current-sensing-1      |     return self.execute(
current-sensing-1      |   File 
"/usr/local/lib/python3.9/site-packages/pymodbus/client/base.py", line 
179, in execute
current-sensing-1      |     raise ConnectionException(f"Not 
connected[{self!s}]")
current-sensing-1      | pymodbus.exceptions.ConnectionException: Modbus 
Error: [Connection] Not connected[AsyncModbusTcpClient 192.168.0.7:502]

Actually I can't find the 3 retires thing in that snippet, the logs are so messy. Ask if you want a few thousand lines more.

tobyaharris commented 1 month ago

Not resolved by sample rate, the above was taken at 5s sample spacing

tobyaharris commented 1 month ago

Try pinging / going to dashboard at 192.168.0.7 etc to test ethernet connection