Modbus #33872 does not work with Serial Devices

HarrisonPace commented 4 years ago

The problem

The new Modbus PR (designed to fix the broken implementation of the 0.108.3 integration), does not work with Serial Devices only TCP/IP devices.

Environment

Home Assistant Core release with the issue: 0.108.3 (with PR https://github.com/home-assistant/core/pull/34043)
Last working Home Assistant Core release (if known): 0.107.7
Operating environment (Home Assistant/Supervised/Docker/venv): Home Assistant
Integration causing this issue: Modbus
Link to integration documentation on our website: https://www.home-assistant.io/integrations/modbus/

Problem-relevant `configuration.yaml`

sensor:
# PLC (Cabinet Sensors)
  - platform: modbus
    scan_interval: 6
    registers:
      # Cabinet Temperature
      - name: PLC Cabinet Temperature
        hub: Skylight Controller
        unit_of_measurement: °C
        register: 0
        scale: 0.01
        precision: 2
      # Cabinet Minimum Temperature
      - name: PLC Cabinet Minimum Temperature
        hub: Skylight Controller
        unit_of_measurement: °C
        register: 1
        scale: 0.01
        precision: 2
      # Cabinet Maximum Temperature
      - name: PLC Cabinet Maximum Temperature
        hub: Skylight Controller
        unit_of_measurement: °C
        register: 2
        scale: 0.01
        precision: 2
      # Cabinet Average Daily Temperature
      - name: PLC Cabinet Daily Average Temperature
        hub: Skylight Controller
        unit_of_measurement: °C
        register: 3
        scale: 0.01
        precision: 2
      # Cabinet Minimum Daily Temperature
      - name: PLC Cabinet Minimum Daily Temperature
        hub: Skylight Controller
        unit_of_measurement: °C
        register: 25
        scale: 0.01
        precision: 2
      # Cabinet Maximum Daily Temperature
      - name: PLC Cabinet Maximum Daily Temperature
        hub: Skylight Controller
        unit_of_measurement: °C
        register: 26
        scale: 0.01
        precision: 2

      # Cabinet Humidity
      - name: PLC Cabinet Humidity
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 10
        scale: 0.01
        precision: 2
      # Cabinet Minimum Humidity
      - name: PLC Cabinet Minimum Humidity
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 11
        scale: 0.01
        precision: 2
      # Cabinet Maximum Humidity
      - name: PLC Cabinet Maximum Humidity
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 12
        scale: 0.01
        precision: 2
      # Cabinet Average Daily Humidity
      - name: PLC Cabinet Daily Average Humidity
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 13
        scale: 0.01
        precision: 2
      # Cabinet Minimum Daily Humidity
      - name: PLC Cabinet Minimum Daily Humidity
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 27
        scale: 0.01
        precision: 2
      # Cabinet Maximum Daily Humidity
      - name: PLC Cabinet Maximum Daily Humidity
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 28
        scale: 0.01
        precision: 2

      # Skylight (Bathroom Window Position)
      - name: Skylight Bathroom Window Position
        hub: Skylight Controller
        unit_of_measurement: Steps
        register: 20
      # Skylight (Ensuite Window Position)
      - name: Skylight Ensuite Window Position
        hub: Skylight Controller
        unit_of_measurement: Steps
        register: 21

      # Skylight (Bathroom Window Position)
      - name: 'Skylight Bathroom Position (%)'
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 22
      # Skylight (Ensuite Window Position)
      - name: 'Skylight Ensuite Position (%)'
        hub: Skylight Controller
        unit_of_measurement: '%'
        register: 23

      # Skylight (Bathroom Operation - On Time)
      - name: Skylight (Bathroom Operation - On Time)
        hub: Skylight Controller
        unit_of_measurement: s
        register: 15

      # Skylight (Ensuite Operation - On Time)
      - name: Skylight (Ensuite Operation - On Time)
        hub: Skylight Controller
        unit_of_measurement: s
        register: 16

      # Skylight (Bathroom Operation - Off Time)
      - name: Skylight (Bathroom Operation - Off Time)
        hub: Skylight Controller
        unit_of_measurement: s
        register: 17

      # Skylight (Ensuite Operation - Off Time)
      - name: Skylight (Ensuite Operation - Off Time)
        hub: Skylight Controller
        unit_of_measurement: s
        register: 18

binary_sensor:
  - platform: modbus
    scan_interval: 2
    inputs:
      - name: High High Temperature - Cabinet Alarm
        hub: Skylight Controller
        slave: 1
        address: 800
      - name: High Temperature - Cabinet Alarm
        hub: Skylight Controller
        slave: 1
        address: 801
      - name: Low Temperature - Cabinet Alarm
        hub: Skylight Controller
        slave: 1
        address: 802
      - name: High High Humidity - Cabinet Alarm
        hub: Skylight Controller
        slave: 1
        address: 816
      - name: High Humidity - Cabinet Alarm
        hub: Skylight Controller
        slave: 1
        address: 817
      - name: Low Humidity - Cabinet Alarm
        hub: Skylight Controller
        slave: 1
        address: 818
      - name: Bathroom Skylight Window - Open 
        hub: Skylight Controller
        slave: 1
        address: 145
      - name: Ensuite Skylight Window - Open 
        hub: Skylight Controller
        slave: 1
        address: 146

#Modbus TCP Connection (Siemens)
modbus:
  - name: Skylight Controller
    type: tcp
    host: 192.168.80.155
    port: 502
    delay: 0
  - name: Curtain Controller
    type: serial
    method: rtu
    port: /dev/serial/by-id/usb-www.freetronics.com_0003_95437313934351505191-if00
    baudrate: 9600
    stopbits: 1
    bytesize: 8
    parity: N
    timeout: 10

    # Modbus - PLC Cabinet
  - platform: modbus
    coils:
    # Climate Stats (Resets)
    - name: "PLC Cabinet Maximum Temperature Reset"
      hub: Skylight Controller
      slave: 1
      coil: 81
    - name: "PLC Cabinet Minimum Temperature Reset"
      hub: Skylight Controller
      slave: 1
      coil: 82
    - name: "PLC Cabinet Daily Temperature Reset"
      hub: Skylight Controller
      slave: 1
      coil: 83
    - name: "PLC Cabinet Maximum Humidity Reset"
      hub: Skylight Controller
      slave: 1
      coil: 84
    - name: "PLC Cabinet Minimum Humidity Reset"
      hub: Skylight Controller
      slave: 1
      coil: 85
    - name: "PLC Cabinet Daily Humidity Reset"
      hub: Skylight Controller
      slave: 1
      coil: 86
    - name: "PLC Cabinet Maximum Daily Temperature Reset"
      hub: Skylight Controller
      slave: 1
      coil: 881
    - name: "PLC Cabinet Minimum Daily Temperature Reset"
      hub: Skylight Controller
      slave: 1
      coil: 882
    - name: "PLC Cabinet Maximum Daily Humidity Reset"
      hub: Skylight Controller
      slave: 1
      coil: 883
    - name: "PLC Cabinet Minimum Daily Humidity Reset"
      hub: Skylight Controller
      slave: 1
      coil: 884

      # Skylight Control
    - name: "PLC Cabinet Service Mode (Remote Activate)"
      hub: Skylight Controller
      slave: 1
      coil: 97

    - name: "Skylight Control - All UP"
      hub: Skylight Controller
      slave: 1
      coil: 98

    - name: "Skylight Control - All Down"
      hub: Skylight Controller
      slave: 1
      coil: 98

      # Individual Skylight Control
    - name: "Skylight Control - Bathroom UP"
      hub: Skylight Controller
      slave: 1
      coil: 113

    - name: "Skylight Control - Bathroom Down"
      hub: Skylight Controller
      slave: 1
      coil: 114

    - name: "Skylight Control - Ensuite UP"
      hub: Skylight Controller
      slave: 1
      coil: 115

    - name: "Skylight Control - Ensuite Down"
      hub: Skylight Controller
      slave: 1
      coil: 116

      # Skylight Time Set
    - name: "Skylight Control - Ensuite On Time Set"
      hub: Skylight Controller
      slave: 1
      coil: 130

    - name: "Skylight Control - Ensuite Off Time Set"
      hub: Skylight Controller
      slave: 1
      coil: 132

    - name: "Skylight Control - Bathroom On Time Set"
      hub: Skylight Controller
      slave: 1
      coil: 129

    - name: "Skylight Control - Bathroom Off Time Set"
      hub: Skylight Controller
      slave: 1
      coil: 131

    - name: " Bedroom Curtain - UP"
      hub: Curtain Controller
      slave: 1
      coil: 0

    - name: " Bedroom Curtain - Down"
      hub: Curtain Controller
      slave: 1
      coil: 1

    - name: " Bedroom Curtain - Stop"
      hub: Curtain Controller
      slave: 1
      coil: 2

Traceback/Error logs

Log Details (WARNING)
Logger: homeassistant.helpers.entity
Source: __main__.py:342
First occurred: 4:22:43 AM (36 occurrences)
Last logged: 4:29:24 AM

Update of switch.bedroom_curtain_up is taking over 10 seconds
Update of switch.bedroom_curtain_down is taking over 10 seconds
Update of switch.bedroom_curtain_stop is taking over 10 seconds

Additional information

Debug Log: https://paste.ubuntu.com/p/M4Y9xNjYxn/

janiversen commented 4 years ago

As noted on discord, please update the issue by only having serial connections in your config. Once done please update config and debug log. Thanks in advance.

HarrisonPace commented 4 years ago

So after testing it seems that simply the Modbus device doesn't respond, so I garner its not been sent out correctly?

2020-04-12 12:38:48 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.

2020-04-12 12:38:49 WARNING (MainThread) [pymodbus.client.asynchronous] Not Importing deprecated clients. Dependency Twisted is not Installed

2020-04-12 12:38:49 DEBUG (MainThread) [pymodbus.client.asynchronous.asyncio] Connecting.

2020-04-12 12:38:50 DEBUG (MainThread) [pymodbus.client.asynchronous.asyncio] Client connected to modbus server

2020-04-12 12:38:50 INFO (MainThread) [pymodbus.client.asynchronous.asyncio] Protocol made connection.

2020-04-12 12:38:50 INFO (MainThread) [pymodbus.client.asynchronous.asyncio] Connected to /dev/serial/by-id/usb-www.freetronics.com_0003_95437313934351505191-if00

2020-04-12 12:39:22 DEBUG (MainThread) [pymodbus.client.asynchronous.asyncio] send: 0x1 0x1 0x0 0x0 0x0 0x1 0xfd 0xca

(full logs: https://paste.ubuntu.com/p/73vYcVP4K6/ )

HarrisonPace commented 4 years ago

As discussed on Discord calling modbus.write_coil results in the service working once every 50 or so times, thus it looks like it may be a timing issue.

probot-home-assistant[bot] commented 4 years ago

Hey there @adamchengtkc, mind taking a look at this issue as its been labeled with a integration (modbus) you are listed as a codeowner for? Thanks!

allmoney commented 4 years ago

I have this error: 2020-04-13_16-04-29

janiversen commented 4 years ago

This is a closed issue, this problem is solved, however we still have a couple of other issues.

However your error look more like a configuration error.

allmoney commented 4 years ago

Before 108 version no errors with configuration... And I see it not closed:

janiversen commented 4 years ago

Well let me put it differently, a patch have been merged into dev for this issue...and discussions etc. are continuing in other issues.

Apart from that it is quite impossible to help you without more information like e.g. your logs and configuration.

marhoy commented 4 years ago

With the following patch to pymodbus-2.3.0:

--- a/pymodbus/client/asynchronous/factory/serial.py
+++ b/pymodbus/client/asynchronous/factory/serial.py
@@ -102,8 +102,6 @@ def async_io_factory(port=None, framer=None, **kwargs):
         sys.exit(1)

     client = AsyncioModbusSerialClient(port, proto_cls, framer, loop, **kwargs)
-    coro = client.connect()
-    loop.run_until_complete(coro)
     return loop, client

I'm able to create a AsyncModbusSerialClient with an already running event loop. (The above run_until_complete() fails when the provided loop is already running. And in HA, we want to use the already running HA event loop).

I can then create a test-script that creates a loop, an AsyncModbusSerialClient, handles the client.connect(), and tries to read an input register from my serial modbus device.

Problem is, it doesn't work consistently: About half of the time, it works as expected. The other times, the script just hangs inside the pymodbus rtu_framer. There seems to be (at least one) bug in the pymodbus asyncio implementation wrt. parsing the received bits.

I'm happy to send the test-script and/or debug logs from it to anyone who's interested.

janiversen commented 4 years ago

I am interested.

lopez3 commented 4 years ago

Hi, I am using modbus with a serial port and it does not work for me either, let me know what information could help you.

marhoy commented 4 years ago

@janiversen I belive this is an issue with the pymodbus library, so I've opened an issue there will all the details of my findings: https://github.com/riptideio/pymodbus/issues/500

It seems that the async support in pymodbus is flaky at best.

I appreciate your effort to update to the latest pymodbus package, but are you sure it's a good idea to also switch to the async modbus client? I haven't had any issues with the sync client.

janiversen commented 4 years ago

I happen to agree with you, and will make a patch that restores the sync client, but keeps all the other changes made.

I will leave the async version committed so it is available when there are a new version of pymodbus.

A maybe better option long term, is only to use the protocol component from pymodbus and handle the socket/serial communication in the integration, which would allow us to use the parts available in core.

Buddha-zen commented 4 years ago

I support. Home Assistant 0.108.4, during the initialization of modbus RTU, the Home Assistant service crashes and there is no access to the site. We are waiting for corrections :)

janiversen commented 4 years ago

The others have not reported a crash of HA, that is something very new !!

We know the modbus integration stops, but that is not HA.

It is quite ok, that you are waiting for corrections, but when you report a problem like a crash of HA, without logs or anything, do not expect a solution to your problem. A HA crash needs a new issue, because in this issue we only work to solve the serial problem, which is in the communication and not a crash.

allmoney commented 4 years ago

Updated to Home Assistant 0.108.4 Same error. In logs found: Error executing script. Service not found for call_service at pos 1: Unable to find service modbus/write_register

In UI I see:

allmoney commented 4 years ago

And more in logs:

Logger: homeassistant.setup
Source: components/modbus/__init__.py:207
First occurred: 12:26:50 PM (1 occurrences)
Last logged: 12:26:50 PM

Error during setup of component modbus
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 171, in _async_setup_component
    hass, processed_config
  File "/usr/src/homeassistant/homeassistant/components/modbus/__init__.py", line 139, in async_setup
    await hass.async_add_executor_job(start_modbus)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/modbus/__init__.py", line 111, in start_modbus
    client.setup()
  File "/usr/src/homeassistant/homeassistant/components/modbus/__init__.py", line 207, in setup
    loop=self._loop,
  File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/serial.py", line 75, in __new__
    yieldable = factory_class(framer=framer, port=port, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/factory/serial.py", line 104, in async_io_factory
    client = AsyncioModbusSerialClient(port, proto_cls, framer, loop, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/asyncio/__init__.py", line 689, in __init__
    self._connected_event = asyncio.Event()
  File "/usr/local/lib/python3.7/asyncio/locks.py", line 249, in __init__
    self._loop = events.get_event_loop()
  File "/usr/local/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'SyncWorker_12'.

iltomma commented 4 years ago

update to 0.108.5

`WARNING (MainThread) [pymodbus.client.asynchronous] Not Importing deprecated clients. Dependency Twisted is not Installed

ERROR (MainThread) [homeassistant.setup] Error during setup of component modbus

Traceback (most recent call last):

File "/usr/src/homeassistant/homeassistant/setup.py", line 171, in _async_setup_component hass, processed_config

File "/usr/src/homeassistant/homeassistant/components/modbus/init.py", line 139, in async_setup await hass.async_add_executor_job(start_modbus)

File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs)

File "/usr/src/homeassistant/homeassistant/components/modbus/init.py", line 111, in start_modbus client.setup()

File "/usr/src/homeassistant/homeassistant/components/modbus/init.py", line 207, in setup loop=self._loop,

File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/serial.py", line 75, in new yieldable = factory_class(framer=framer, port=port, **kwargs)

File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/factory/serial.py", line 104, in async_io_factory client = AsyncioModbusSerialClient(port, proto_cls, framer, loop, **kwargs)

File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/asyncio/init.py", line 689, in init self._connected_event = asyncio.Event()

File "/usr/local/lib/python3.7/asyncio/locks.py", line 249, in init self._loop = events.get_event_loop()

File "/usr/local/lib/python3.7/asyncio/events.py", line 644, in get_event_loop % threading.current_thread().name)

RuntimeError: There is no current event loop in thread 'SyncWorker_3'.

ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform modbus.sensor: Unable to set up component.`

and

lemiks commented 4 years ago

update to 0.108.5

`WARNING (MainThread) [pymodbus.client.asynchronous] Not Importing deprecated clients. Dependency Twisted is not Installed

ERROR (MainThread) [homeassistant.setup] Error during setup of component modbus

Traceback (most recent call last):

File "/usr/src/homeassistant/homeassistant/setup.py", line 171, in _async_setup_component hass, processed_config

File "/usr/src/homeassistant/homeassistant/components/modbus/init.py", line 139, in async_setup await hass.async_add_executor_job(start_modbus)

File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs)

File "/usr/src/homeassistant/homeassistant/components/modbus/init.py", line 111, in start_modbus client.setup()

File "/usr/src/homeassistant/homeassistant/components/modbus/init.py", line 207, in setup loop=self._loop,

File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/serial.py", line 75, in new yieldable = factory_class(framer=framer, port=port, **kwargs)

File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/factory/serial.py", line 104, in async_io_factory client = AsyncioModbusSerialClient(port, proto_cls, framer, loop, **kwargs)

File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/asyncio/init.py", line 689, in init self._connected_event = asyncio.Event()

File "/usr/local/lib/python3.7/asyncio/locks.py", line 249, in init self._loop = events.get_event_loop()

File "/usr/local/lib/python3.7/asyncio/events.py", line 644, in get_event_loop % threading.current_thread().name)

RuntimeError: There is no current event loop in thread 'SyncWorker_3'.

ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform modbus.sensor: Unable to set up component.`

and

The same log on HA started 0.108.0 and continues till last 0.108.5

vdemidov commented 4 years ago

2020-04-16 00:44:40 WARNING (MainThread) [pymodbus.client.asynchronous] Not Importing deprecated clients. Dependency Twisted is not Installed
2020-04-16 00:44:43 ERROR (MainThread) [homeassistant.setup] Error during setup of component modbus
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 171, in _async_setup_component
    hass, processed_config
  File "/usr/src/homeassistant/homeassistant/components/modbus/__init__.py", line 139, in async_setup
    await hass.async_add_executor_job(start_modbus)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/modbus/__init__.py", line 111, in start_modbus
    client.setup()
  File "/usr/src/homeassistant/homeassistant/components/modbus/__init__.py", line 207, in setup
    loop=self._loop,
  File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/serial.py", line 75, in __new__
    yieldable = factory_class(framer=framer, port=port, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/factory/serial.py", line 104, in async_io_factory
    client = AsyncioModbusSerialClient(port, proto_cls, framer, loop, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pymodbus/client/asynchronous/asyncio/__init__.py", line 689, in __init__
    self._connected_event = asyncio.Event()
  File "/usr/local/lib/python3.7/asyncio/locks.py", line 249, in __init__
    self._loop = events.get_event_loop()
  File "/usr/local/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'SyncWorker_5'.
2020-04-16 00:44:49 ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform modbus.sensor: Unable to set up component.

Errors in HA from 0.108.1 to 0.108.5 version.

HarrisonPace commented 4 years ago

So anyone who is having issues I recommend simply copying the 0.107.7 modbus implementation into a 0.108.x installation until this is fixed.

This method works fine, however I would recommend implementation is changed back until the serial component can be fixed or pymodbus is implemented using a different serial method in HASS. Its less than ideal to have a broken implementation across several release cycles (even if this is a result of upstream API issue).

janiversen commented 4 years ago

I am preparing that as we speak.....however I want to keep the other features added, so it is not just a simple copy/paste operation. I miss updating 1 file and test before making the pr, for dome of you to test

janiversen commented 4 years ago

I hope the above PR solves the issue.

@thehaxxa you have a big configuration, before merging the PR I would really like to hear if it works for you. Thanks in advance.

Of course the more that test the PR, the more secure we can be that the issue is gone.

HarrisonPace commented 4 years ago

@thehaxxa you have a big configuration, before merging the PR I would really like to hear if it works for you. Thanks in advance.

All good from my side, everything is working.

I get this in my logs but everything works:

Source: __main__.py:342
First occurred: 1:22:15 PM (2 occurrences)
Last logged: 1:22:25 PM

Update of switch.bedroom_curtain_up is taking over 10 seconds
Update of switch.bedroom_curtain_down is taking over 10 seconds

Cleanup recv buffer before send: 0x54 0x55 0x20 0x43 0x75 0x72 0x74 0x61 0x69 0x6e 0x20 0x53 0x65 0x72 0x76 0x65 0x72 0xd 0xa 0x1 0x1 0x1 0x0 0x51 0x88
Connection lost. Reconnecting…

Buddha-zen commented 4 years ago

Installed version 0.108.7, ModBus RTU works with only one slave. If there are 2 or more slaves, data is read only from the first in the configuration list. If someone has more than one slave, check if it works ...

janiversen commented 4 years ago

This is a closed issue. It sounds like you might have a new problem, so you need to create a new issue.

Please do not forget to add:

a minimal configuration showing the problem, with the following logger added: ‘’’ logger: logs: homeassistant.components.modbus: debug pymodbus.client: debug ‘’’

And of course the log from a run with logger active.

Without that we are not really able to help.

home-assistant / core