Extension not working - probably need some clarification in documentation?

[kepi]

This is probably some misconfiguration on my part, but I'm not able to find what is wrong in logs or documentation.

Worth to mention that I added yml for xS10 extension myself #121 so there might be some error with it? I'm not sure at this moment how unipi-control decides which hw definition is used.

Here is relevant part of my configuration:

modbus_serial:
  port: /dev/extcomm/0/0
  baud_rate: 19200
  units:
    - unit: 1
      device_name: xS10
      identifier: xS10
      suggested_area: 1.03 Technická

Btw. I can't find in docs what identifier is used for.

In evok, working configuration is:

[EXTENSION_1]
global_id = 2
device_name = xS10
modbus_uart_port = /dev/extcomm/0/0
allow_register_access = False
address = 1
scan_frequency = 10
scan_enabled = True 
baud_rate = 19200  

Evok is stopped. When I start unipi-control, it says Reading extensions but there are no features from extension available, only those from main unit are.

(.venv) unipi@holly:~/dev/unipi_control$ sudo /home/unipi/.pyenv/shims/unipi-control
2023-11-19 21:05:54,959 | INFO | [MODBUS] TCP client connected to localhost:502
2023-11-19 21:05:54,963 | INFO | [MODBUS] Serial client connected to /dev/extcomm/0/0
2023-11-19 21:05:54,964 | INFO | [MODBUS] Reading SPI boards
2023-11-19 21:05:54,979 | INFO | [MODBUS] Reading extensions
2023-11-19 21:05:56,010 | INFO | [CONFIG] 72 features initialized.

Thanks for any tips.

[mh-superbox]

Hello, as I do not own any extensions myself, this has not yet been implemented. I have already made preparations for Modbus RTU, as I operate some electricity meters via Modbus RTU.

This is the place where the extensions are initialized: https://github.com/superbox-dev/unipi-control/blob/856d0121d3f4bf4d3d8c38343f961119861786ab/unipi_control/neuron.py#L270

The following steps are necessary:

A new class is required at https://github.com/superbox-dev/unipi-control/tree/main/unipi_control/extensions

This class is required to be able to address all features (RO, DI, ...) of the extensions via Modbus RTU.

The support of extensions has always been a big concern for me. If you have a suitable code for this, I can support you as far as I can.

[kepi]

I tried to implement this today, you can see WIP in mentioned branch above.

Problem is, that I'm getting

2023-11-20 21:33:28,736 | ERROR | [MODBUS] Modbus Error: [Input/Output] ERROR: No response received after 3 retries
2023-11-20 21:33:28,737 | ERROR | [MODBUS] Modbus Error: [Connection] Not connected[AsyncModbusSerialClient /dev/extcomm/0/0:0]

When trying to scan serial in mqtt features. It works fine before, I'm getting firmware version, with previous attempt I also got list of features published as name_ext1 successfully. Setting value through mqtt worked, but getting it not (that is when i realized i have to add support to mqtt/features).

Some design decisions I'm not sure you will like:

1. as extension is quite same as PLC and has same features, I used features/neuron
2. i split Board coded from neuron to unipi_device.py (naming improvement would be great :D)
3. i added some logic extension/neuron in neuron features so correct tcp/serial modbus is called

At this point, I'm just trying to get it working but feel free to suggest any changes, ideas etc.

If you had any idea where to look for problem with serialclient or how to debug this, let me know. I'm not sure how much time I'll have in next days, but I would like to work on this when it is fresh.

TBH - on projects I don't know well, like this one, I often simply try to implement code on best first guess without really going through whole codebase. So I might be missing something completely basic.

Thanks

[mh-superbox]

I like your design decisions. Avoiding duplicates is always good.

I tried your branch on my Unipi and now have the same problem with my Modbus RTU devices.

It is very late today. I'll have another look tomorrow. I'll read up a bit more on the RTU/baudrate topic.

It will be an exciting topic to get the data from several extensions in the shortest possible interval. Modbus RTU has its limitations compared to Modbus TCP.

I am confident that we can solve this :)

[mh-superbox]

Good morning kepi,

can you please execute this script (with your settings and modbus register) and post the output. The error message is normally displayed if the device cannot be reached, e.g. incorrect baud rate or unit id.

from pymodbus.client.serial import ModbusSerialClient
from pymodbus import pymodbus_apply_logging_config

pymodbus_apply_logging_config()

def main():
    client = ModbusSerialClient(
        port="/dev/extcomm/0/0",
        baudrate=9600,
        timeout=1,
        retries=3,
        retry_on_empty=True,
    )
    client.connect()

    try:
        rr = client.read_input_registers(0, 2, slave=1)
    except Exception as e:
        print("ERROR", e)
    else:
        print("RESPONSE", rr.registers)

    client.close()

if __name__ == "__main__":
    main()

[kepi]

Sure,

.venv) unipi@holly:~/dev/unipi_control$ python ~/test.py
2023-11-23 14:47:36,356 DEBUG logging:102 Current transaction state - IDLE
2023-11-23 14:47:36,357 DEBUG logging:102 Running transaction 1
2023-11-23 14:47:36,357 DEBUG logging:102 SEND: 0x1 0x4 0x0 0x0 0x0 0x2 0x71 0xcb
2023-11-23 14:47:36,358 DEBUG logging:102 New Transaction state "SENDING"
2023-11-23 14:47:36,359 DEBUG logging:102 Changing transaction state from "SENDING" to "WAITING FOR REPLY"
2023-11-23 14:47:36,461 DEBUG logging:102 Changing transaction state from "WAITING FOR REPLY" to "PROCESSING REPLY"
2023-11-23 14:47:36,462 DEBUG logging:102 RECV: 0x1 0x4 0x4 0x0 0x0 0x0 0x0 0xfb 0x84
2023-11-23 14:47:36,463 DEBUG logging:102 Processing: 0x1 0x4 0x4 0x0 0x0 0x0 0x0 0xfb 0x84
2023-11-23 14:47:36,464 DEBUG logging:102 Getting Frame - 0x4 0x4 0x0 0x0 0x0 0x0
2023-11-23 14:47:36,465 DEBUG logging:102 Factory Response[ReadInputRegistersResponse': 4]
2023-11-23 14:47:36,466 DEBUG logging:102 Frame advanced, resetting header!!
2023-11-23 14:47:36,466 DEBUG logging:102 Adding transaction 1
2023-11-23 14:47:36,467 DEBUG logging:102 Getting transaction 1
2023-11-23 14:47:36,467 DEBUG logging:102 Changing transaction state from "PROCESSING REPLY" to "TRANSACTION_COMPLETE"
RESPONSE [0, 0]

[mh-superbox]

Okay, you can read the register from this unit, pymodbus works... How much extensions are you connected to the plc?

I want to write a test script to see in what time I can read all registers from the extensions, then we can see how fast this will work. I think Modbus RTU will not be quite as fast as the Modbus TCP registers.

[kepi]

Only single extension. If you need more, I should have somewhere one more, I just didn't need it so far.

[mh-superbox]

I have found the problem. Modbus RTU processes the data sequentially. But we have simultaneous requests. Whenever this happens, this error occurs.

I am trying to find the best solution. I will come up with a solution in the next few days.

The difficulty is that the data should be requested as quickly as possible to avoid a long delay with digital inputs or relays.

A test with several devices via Modbus RTU makes sense in any case.

Because I have mainly used Modbus TCP up to now, I have not yet had this problem.

[kepi]

Next few days would be amazing. I'll try to find and install second extension so I'm ready for tests, should have time during weekend. Thanks!

[mh-superbox]

I am currently working on the code modifications. Since some of my aiomqtt code is deprecated, I am making these necessary modifications at the same time.

I have already created a branch (still without extensions). However, I am not yet happy with the speed of Modbus TCP/RTU.

I will do further tests. I don't know if I'll be able to do it all by the weekend.

https://github.com/superbox-dev/unipi-control/tree/dev/refactor

Which Python version are you using? I will test a few asyncio functions that are available since Python 3.9.

[mh-superbox]

Hello kepi,

I have another test for you. Can you please run these two scripts. I would like to test how quickly I can retrieve the extension registers.

import asyncio
import contextlib
from time import time

from pymodbus.client import AsyncModbusSerialClient

units = [
    1,
]

registers = [
    # DI 1.x / RO 1.x
    {"address": 0, "count": 2},
]

async def main():
    client = AsyncModbusSerialClient(port="/dev/extcomm/0/0", baudrate=18200)

    while True:
        start1 = time()

        if not client.connected:
            print("connect")
            await client.connect()

        for unit in units:
            start2 = time()

            for register in registers:
                response = await client.read_input_registers(**register, slave=unit)
            await asyncio.sleep((3.5 * 11 / 18200) * 2)

            print("UNIT", unit, time() - start2)
        print("TOTAL", time() - start1)

if __name__ == "__main__":
    with contextlib.suppress(KeyboardInterrupt):
        asyncio.run(main())

from time import sleep
from time import time

import minimalmodbus

units = [
    1,
]

registers = [
    # DI 1.x / RO 1.x
    {"address": 0, "count": 2},
    # LED
    {"address": 31, "count": 1},
]

def main():
    while True:
        start1 = time()
        for unit in units:
            instrument = minimalmodbus.Instrument("/dev/extcomm/0/0", unit, debug=False)
            instrument.serial.baudrate = 18200
            start2 = time()
            # wait 3.5 chars * 11 bits / baudrate
            sleep(3.5 * 11 / 18200)

            for register in registers:
                try:
                    response = instrument.read_registers(register["address"], register["count"], functioncode=4)
                except minimalmodbus.InvalidResponseError as e:
                    print(unit, e)
                except minimalmodbus.NoResponseError as e:
                    print(unit, e)

            instrument.serial.close()
            print("UNIT", time() - start2)

        print("TOTAL", time() - start1)

if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        ...

[kepi]

Hi,

it's completely OK if it won't be ready by weekend. I just ment that I might have time during weekend, not trying to rush you. On the other hand, I'm really looking forward to the moment when I'll be able to switch to superbox :)

Python - 3.11.4

Attaching output from tests:

• test2.output.txt
• test3.output.txt

timeout 60s python ./test2.py > test2.output.txt ; timeout 60s python ./test3.py > test3.output.txt

In test3 I changed baud rate in both places, in serial.baudrate and then in sleep.

[mh-superbox]

The test2.output.txt is the pymodbus test? I have updated the pymodbus test (added an asyncio.sleep). Please run it again.

[kepi]

yes, here is new result with asyncio. Btw. it works in batches now, always get couple connects fast, then probably sleep, then some more etc.

that pause isn't visible in timings.

test4.output.txt

[mh-superbox]

The sleep is the 3.5 chars time, without this sleep all await client.read_input_registers(**register, slave=unit) are slow. I have found the bug. The 3.5 chars time is required at the begin and end of a modbus rtu request. I updated the script.

More info: https://minimalmodbus.readthedocs.io/en/stable/serialcommunication.html

[kepi]

My bad, it isn't sleep. Simply with pymodbus test, communication is super slow and really inconsistent. It takes often 3 seconds to complete at completely random places, as you can see in the output.

Minimalmodbus test is super fast, but getting some checkusm errors. I tried to play with it for couple minutes, but no quick discovery so far.

Btw. according to docs you mentioned, shouldn't there be the sleep after every read input registers? I'm not really sure, just asking. To be sure I tried it in both scripts and don't see any meaningfull difference.

[mh-superbox]

My bad, it isn't sleep. Simply with pymodbus test, communication is super slow and really inconsistent. It takes often 3 seconds to complete at completely random places, as you can see in the output.

Minimalmodbus test is super fast, but getting some checkusm errors. I tried to play with it for couple minutes, but no quick discovery so far.

Btw. according to docs you mentioned, shouldn't there be the sleep after every read input registers? I'm not really sure, just asking. To be sure I tried it in both scripts and don't see any meaningfull difference.

Yes, I have also noticed that pymodbus is not really the fastest library, but it is very stable compared to minimalmodbus. I also have these CRC errors with minimalmodbus on my electricity meters. So it's not because of the extensions.

It looks like the 3.5 chars are only necessary once between each request. The problem may be with async. Can you run the pymodbus test without async?

Info: https://ozeki.hu/p_5854-modbus-rtu.html

btw. I found this issue: https://github.com/pymodbus-dev/pymodbus/pull/1810 but this fix is only in the dev branch.

[kepi]

Here are results for 60 seconds run for each test. I run one sync pymodbus with patch mentioned in pymodbus-dev/pymodbus#1810

unipi@holly:~/tests$ grep -c TOTAL output*
output_async.txt:209
output_minmodbus.txt:1622
output_sync.txt:156
output_sync_patched.txt:311

[koutnada]

Hi guys, I've been following your communication since it began, and I've recently been able to run some tests myself. I have two xS11 devices connected to a Neuron L203, and I've successfully read registers from the extensions using mbpoll to ensure the correct settings and addresses.

I've been debugging the unipi_control code, and I've reached a point where Modbus Serial is attempting to acquire a response from the extension. However, on the first attempt, the Modbus client successfully connects but doesn't receive a response, which likely leads to the client disconnecting. Subsequent attempts result in the following error:

2024-01-06 09:47:33,633 | ERROR | [MODBUS] Modbus Error: [Connection] Connection lost during request 2024-01-06 09:47:33,636 | ERROR | [MODBUS] Modbus Error: [Connection] Not connected[AsyncModbusSerialClient /dev/extcomm/0/0:0]

I came across this topic on GitHub: https://github.com/pymodbus-dev/pymodbus/issues/629 which led me to a potential solution here: https://github.com/pymodbus-dev/pymodbus/pull/631

I plan to spend some more time debugging and attempting to maintain the client connection.

Thanks for you effort, I'm looking forward for the solution of this problem, as it looks like it is close. Finally I will be able to switch my lights using MQTT :-).

Adam