search

brainelectronics / micropython-modbus

MicroPython Modbus RTU Slave/Master and TCP Server/Slave library
GNU General Public License v3.0
104 stars 45 forks source link

Add asyncio support #56

Open GimmickNG opened 1 year ago

GimmickNG commented 1 year ago

With reference to my comment in #5, I've added asyncio support for TCP and limited support for Serial RTU. Right now, async serial support is TBD but I've left the class in just in case anyone else wants to try modifying it themselves. To avoid code duplication, I've based the async classes off their synchronous equivalents, and only modified parts when I couldn't cleanly link the two.

I haven't tested the results yet, but I expect it to work for the most part. So I guess until later, this is still very much as "use at your own risk" situation since it could be rather buggy now.

brainelectronics commented 1 year ago

Thank you @GimmickNG for your PR 🎊

I need a deeper review and testing the next days but the first impression is very good 👍🏻

brainelectronics commented 1 year ago

Please mention #5 and #11 in the Changelog entry

GimmickNG commented 1 year ago

Thanks for the review. I'll make the changes required (and I also spotted some errors when I tried using the library, whose fixes I'll add here) in a week or so, when I should have more time available.

GimmickNG commented 1 year ago

I'm not sure why Github doesn't recognize that the async folder was renamed to asynchronous in my latest revision. Oh well. Here are the changes to save time on reviewing:

beyonlo commented 1 year ago

Hello. I'm happy that we are closer to have the async feature ready on the ModBus lib :partying_face:

I would like to say that I will happy to help on the tests with the serial version (RTU) that require a board with UART. I have one board with RS485 already working! As soon this PR will be merged I can to help do the tests :smiley:

brainelectronics commented 1 year ago

@beyonlo could you test the first release candidate of this lib with Async support?

https://test.pypi.org/project/micropython-modbus/2.4.0rc39.dev56/

beyonlo commented 1 year ago

Hello @brainelectronics

@beyonlo could you test the first release candidate of this lib with Async support?

Yes, of course I can do that.

Are there some example to run the Slave TCP and Slave RTU using the uasyncio, or a simple doc how to use/start it?

https://test.pypi.org/project/micropython-modbus/2.4.0rc39.dev56/

I downloaded this version, but it seems not to have the uasyncio support. 

pi@pi:~/Downloads/micropython-modbus-2.4.0rc39.dev56$ ls -la
total 32
drwxrwxr-x 4 pi pi  4096 mar  6 10:38 .
drwxr-xr-x 5 pi pi 20480 mar  6 10:38 ..
drwxrwxr-x 2 pi pi  4096 mar  6 10:38 micropython_modbus.egg-info
drwxrwxr-x 2 pi pi  4096 mar  6 10:46 umodbus
pi@pi:~/Downloads/micropython-modbus-2.4.0rc39.dev56$ cd umodbus/
pi@pi:~/Downloads/micropython-modbus-2.4.0rc39.dev56/umodbus$ ls -la
total 136
drwxrwxr-x 2 pi pi  4096 mar  6 10:46 .
drwxrwxr-x 4 pi pi  4096 mar  6 10:38 ..
-rw-r--r-- 1 pi pi 14741 mar  6 05:04 common.py
-rw-r--r-- 1 pi pi  6785 mar  6 05:04 const.py
-rw-r--r-- 1 pi pi 14778 mar  6 05:04 functions.py
-rw-r--r-- 1 pi pi   108 mar  6 05:04 __init__.py
-rw-r--r-- 1 pi pi 36148 mar  6 05:04 modbus.py
-rw-r--r-- 1 pi pi 17275 mar  6 05:04 serial.py
-rw-r--r-- 1 pi pi  1304 mar  6 05:04 sys_imports.py
-rw-r--r-- 1 pi pi 15704 mar  6 05:04 tcp.py
-rw-r--r-- 1 pi pi  2121 mar  6 05:04 typing.py
-rw-r--r-- 1 pi pi   140 mar  6 05:07 version.py
pi@pi:~/Downloads/micropython-modbus-2.4.0rc39.dev56/umodbus$ cat version.py 
#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

__version_info__ = ("2", "4", "0")
__version__ = '.'.join(__version_info__) + '-rc39.dev56'
pi@pi:~/Downloads/micropython-modbus-2.4.0rc39.dev56/umodbus$ grep -i uasyncio *
pi@pi:~/Downloads/micropython-modbus-2.4.0rc39.dev56/umodbus$

Thank you!

GimmickNG commented 1 year ago

@beyonlo I am currently writing a few examples for the async clients and servers. However, I noticed that when writing the client and server examples for async RTU, some features in TCP were not present in RTU and had to be implemented manually (such as serve_forever()). Also, starting an async RTU client was made confusing since it required creating an AsyncRTUServer instance, which makes no sense to the end user.

@brainelectronics I am making changes to make the async version more consistent with the sync one for the RTU implementation - namely, changing the regular CommonRTUFunctions to RTUServer and using this to create an AsyncRTU (client) and a proper AsyncRTUServer class, which will be used by AsyncModbusRTU to start an RTU server. What are your thoughts on this, with respect to e.g. backwards compatibility?

brainelectronics commented 1 year ago

@beyonlo best finding so far 😂 I've fixed the packaging code, let's try again with 2.4.0rc40.dev56

@GimmickNG could you please update the setup.py as follows

    packages=['umodbus', 'umodbus/asynchronous'],

Regarding backwards compability: As long as the following imports and the top level function names do not change in the sync part of the lib, I'm open to many changes as long as they are related to this PR of course

from umodbus.serial import ModbusRTU
from umodbus.serial import Serial as ModbusRTUMaster
from umodbus.tcp import ModbusTCP
from umodbus.tcp import TCP as ModbusTCPMaster
beyonlo commented 1 year ago

@GimmickNG Thanks for your effort to support uasyncio on this lib. I will wait your changes on the uasyncio version, as well the examples for the uasyncio clients and servers to start the tests!

Will be all features of this lib available/works on the uasyncio version too, including the callbacks functions when Master set/get the address on the Slaves?

GimmickNG commented 1 year ago

@beyonlo Since the async version is based off/depends on the sync version, the two should be fairly similar (with the exception of asyncio.run() needed to start the asyncio loop), so yes the callbacks should be supported --- let me know if they aren't.

Also, check async_examples.py for the async TCP/RTU examples. I haven't verified these though so there may be some errors, but I don't expect any initially.

@brainelectronics I split off the server/slave functionality of Serial into a separate RTUServer class since it looked overloaded in comparison to TCP/TCPServer. I tried to keep the user-facing interface as close to the original as possible, so the ModbusRTU class remains the same --- it now uses RTUServer instead of Serial, and Serial is now an RTU client only.

beyonlo commented 1 year ago

Hello @GimmickNG

@beyonlo Since the async version is based off/depends on the sync version, the two should be fairly similar (with the exception of asyncio.run() needed to start the asyncio loop), so yes the callbacks should be supported --- let me know if they aren't.

Understood!

Also, check async_examples.py for the async TCP/RTU examples. I haven't verified these though so there may be some errors, but I don't expect any initially.

I followed your examples to do a simple test running just the Slave TCP using the 2.4.0rc40.dev56, but I have errors. I think that reason is because you did that changes and waiting for the @brainelectronics review and to create a new package.

start_tcp_server.py:

try:
    import uasyncio as asyncio
except ImportError:
    import asyncio

from umodbus.asynchronous.tcp import AsyncModbusTCP, AsyncTCP
from umodbus.asynchronous.serial import AsyncModbusRTU, AsyncSerial

async def start_tcp_server(host, port, backlog):
    server = AsyncModbusTCP()
    await server.bind(local_ip=host, local_port=port, max_connections=backlog)
    await server.serve_forever()

def run_tcp_test(host, port, backlog):
    asyncio.run(start_tcp_server(host, port, backlog))

run_tcp_test('192.168.43.35', 502, 2)
$ mpremote run start_tcp_server.py
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
  File "umodbus/asynchronous/tcp.py", line 21, in <module>
  File "umodbus/asynchronous/modbus.py", line 20, in <module>
  File "umodbus/modbus.py", line 27, in <module>
  File "umodbus/modbus.py", line 672, in Modbus
TypeError: function takes 0 positional arguments but 1 were given

Question: I need to run the Slave TCP and Slave RTU simultaneously in my ESP32-S3. So, how I do to start both (start_tcp_server and start_rtu_server) if just one await server.serve_forever() can to run at same time?

GimmickNG commented 1 year ago

@beyonlo 

$ mpremote run start_tcp_server.py
Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
  File "umodbus/asynchronous/tcp.py", line 21, in <module>
  File "umodbus/asynchronous/modbus.py", line 20, in <module>
  File "umodbus/modbus.py", line 27, in <module>
  File "umodbus/modbus.py", line 672, in Modbus
TypeError: function takes 0 positional arguments but 1 were given

Thanks for letting me know. I'll try with my local version and get back to you later if it's because of the old version or if it's an error in the new code as well.

Question: I need to run the Slave TCP and Slave RTU simultaneously in my ESP32-S3. So, how I do to start both (start_tcp_server and start_rtu_server) if just one await server.serve_forever() can to run at same time?

I'm not sure, but from what I've seen elsewhere, I don't think you need await server.serve_forever() if you are running multiple servers. The serve_forever() seems to be made to ensure that the server task is not cancelled by the program finishing everything else prematurely - so you could (in theory) just have bind(), and then process your other server functions. Something like this (untested):

try:
    import uasyncio as asyncio
except ImportError:
    import asyncio

from umodbus.asynchronous.tcp import AsyncModbusTCP, AsyncTCP
from umodbus.asynchronous.serial import AsyncModbusRTU, AsyncSerial

async def start_tcp_server(host, port, backlog):
    server = AsyncModbusTCP()
    await server.bind(local_ip=host, local_port=port, max_connections=backlog)
    return server

async def start_rtu_server(addr, **kwargs):
    server = AsyncModbusRTU(addr, **kwargs)
    return server

async def run_both():
    tcp_server = await start_tcp_server()
    rtu_server = await start_rtu_server()
    # either (if you are running 2 servers and also some extra tasks)
    while True:
        # other tasks here
        await rtu_server.process()
    # and then cleanup tcp server once application exits
    tcp_server.server_close()

    # or (if only the servers are running)
    # await asyncio.gather(tcp_server.serve_forever(), rtu_server.serve_forever())

asyncio.run(run_both())
brainelectronics commented 1 year ago

This is the next release candidate out for testing 😄 Thank you @beyonlo for testing and thank you @GimmickNG for providing

https://test.pypi.org/project/micropython-modbus/2.4.0rc42.dev56/

beyonlo commented 1 year ago

This is the next release candidate out for testing smile Thank you @beyonlo for testing and thank you @GimmickNG for providing

https://test.pypi.org/project/micropython-modbus/2.4.0rc42.dev56/

Hi all :)

Just to notice you, this version still have the same error:

start_tcp_server.py:

try:
    import uasyncio as asyncio
except ImportError:
    import asyncio

import os

print(f'MicroPython version: {os.uname()}')
from umodbus import version
print(f'Running ModBus version: {version.__version__}')

from umodbus.asynchronous.tcp import AsyncModbusTCP, AsyncTCP
from umodbus.asynchronous.serial import AsyncModbusRTU, AsyncSerial

async def start_tcp_server(host, port, backlog):
    server = AsyncModbusTCP()
    await server.bind(local_ip=host, local_port=port, max_connections=backlog)
    await server.serve_forever()

def run_tcp_test(host, port, backlog):
    asyncio.run(start_tcp_server(host, port, backlog))

run_tcp_test('192.168.43.143', 502, 2)

Output:

$ mpremote run start_tcp_server.py 
MicroPython version: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Running ModBus version: 2.4.0-rc42.dev56
Traceback (most recent call last):
  File "<stdin>", line 12, in <module>
  File "umodbus/asynchronous/tcp.py", line 21, in <module>
  File "umodbus/asynchronous/modbus.py", line 20, in <module>
  File "umodbus/modbus.py", line 27, in <module>
  File "umodbus/modbus.py", line 672, in Modbus
TypeError: function takes 0 positional arguments but 1 were given
brainelectronics commented 1 year ago

@beyonlo thanks for checking, I get the same result on my board locally.

I just merged #64 to run the Docker based tests also on external PRs, could you @GimmickNG update your branch to get this changes in and then have the tests running as well.

GimmickNG commented 1 year ago

@brainelectronics sure. I think the reason for the error (assuming line 672 of modbus.py is correct) is probably because of the overload decorator in typing.py. I didn't get that error during testing because I was using the typing library of pycopy, not the file. I'll change the imports to use the local typing.py import and test to see if I encounter that error.

GimmickNG commented 1 year ago

Looks like it was indeed because of the error with the @overload decorator in typing.py. The async_examples should work now - I managed to get the basic tcp client and server test running on my machine.

beyonlo commented 1 year ago

Hello @GimmickNG! As @brainelectronics do not created a new package with your modification, and I see that your changes is just the + def overload(fun): (Am I right?), so I changed that.

Well, after I changed that on the https://test.pypi.org/project/micropython-modbus/2.4.0rc42.dev56/, the example below run without error now :) Sorry for my ignorance, but I have no idea how to integrate the register definitions, callbacks, etc to run with the async version. With the sync version there is the client.setup_registers(registers=register_definitions) that call the register_definitions, but how that is called by async version? Bellow there is as well a very simple sync example of tcp_client_example.py (based from the tcp_client_example.py ) that I trying to port to use with async version. Could you please, help me how to port that or point me a example of async version using the register_definitions?

My intention is start testing the ModBus TCP Slave and after ModBus RTU Slave, and finally the Masters RTU and TCP. All in async version :)

Thank you so much!

start_tcp_server.py: (async ModBus TCP Slave)

try:
    import uasyncio as asyncio
except ImportError:
    import asyncio

import os

print(f'MicroPython version: {os.uname()}')
from umodbus import version
print(f'Running ModBus version: {version.__version__}')

from umodbus.asynchronous.tcp import AsyncModbusTCP, AsyncTCP
from umodbus.asynchronous.serial import AsyncModbusRTU, AsyncSerial

async def start_tcp_server(host, port, backlog):
    server = AsyncModbusTCP()
    await server.bind(local_ip=host, local_port=port, max_connections=backlog)
    await server.serve_forever()

def run_tcp_test(host, port, backlog):
    asyncio.run(start_tcp_server(host, port, backlog))

run_tcp_test('192.168.43.143', 502, 2)

Output: 

$ mpremote run start_tcp_server.py 
MicroPython version: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Running ModBus version: 2.4.0-rc42.dev56

tcp_client_example.py: (sync ModBus TCP Slave)

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

"""
Main script

Do your stuff here, this file is similar to the loop() function on Arduino

Create a Modbus TCP client (slave) which can be requested for data or set with
specific values by a host device.

The TCP port and IP address can be choosen freely. The register definitions of
the client can be defined by the user.
"""

from umodbus import version
print('Running ModBus version: {}'.format(version.__version__))

# system packages
import time

# import modbus client classes
from umodbus.tcp import ModbusTCP

IS_DOCKER_MICROPYTHON = False
try:
    import network
except ImportError:
    IS_DOCKER_MICROPYTHON = True
    import json

# ===============================================
if IS_DOCKER_MICROPYTHON is False:
    # connect to a network
    station = network.WLAN(network.STA_IF)
    if station.active() and station.isconnected():
        station.disconnect()
        time.sleep(1)
    station.active(False)
    time.sleep(1)
    station.active(True)

    # station.connect('SSID', 'PASSWORD')
    station.connect('Aqui', 'jupi6124')
    time.sleep(1)

    while True:
        print('Waiting for WiFi connection...')
        if station.isconnected():
            print('Connected to WiFi.')
            print(station.ifconfig())
            break
        time.sleep(2)

# ===============================================
# TCP Slave setup
tcp_port = 502              # port to listen to

if IS_DOCKER_MICROPYTHON:
    local_ip = '172.24.0.2'     # static Docker IP address
else:
    # set IP address of the MicroPython device explicitly
    # local_ip = '192.168.4.1'    # IP address
    # or get it from the system after a connection to the network has been made
    local_ip = station.ifconfig()[0]

# ModbusTCP can get TCP requests from a host device to provide/set data
client = ModbusTCP()
is_bound = False

# check whether client has been bound to an IP and port
is_bound = client.get_bound_status()

if not is_bound:
    client.bind(local_ip=local_ip, local_port=tcp_port)

def my_coil_set_cb(reg_type, address, val):
    print('Custom callback, called on setting {} at {} to: {}'.
          format(reg_type, address, val))

def my_coil_get_cb(reg_type, address, val):
    print('Custom callback, called on getting {} at {}, currently: {}'.
          format(reg_type, address, val))

def my_holding_register_set_cb(reg_type, address, val):
    print('Custom callback, called on setting {} at {} to: {}'.
          format(reg_type, address, val))

def my_holding_register_get_cb(reg_type, address, val):
    print('Custom callback, called on getting {} at {}, currently: {}'.
          format(reg_type, address, val))

def my_discrete_inputs_register_get_cb(reg_type, address, val):
    print('Custom callback, called on getting {} at {}, currently: {}'.
          format(reg_type, address, val))

def my_inputs_register_get_cb(reg_type, address, val):
    # usage of global isn't great, but okay for an example
    global client

    print('Custom callback, called on getting {} at {}, currently: {}'.
          format(reg_type, address, val))

    # any operation should be as short as possible to avoid response timeouts
    new_val = val[0] + 1

    # It would be also possible to read the latest ADC value at this time
    # adc = machine.ADC(12)     # check MicroPython port specific syntax
    # new_val = adc.read()

    client.set_ireg(address=address, value=new_val)
    print('Incremented current value by +1 before sending response')

def reset_data_registers_cb(reg_type, address, val):
    # usage of global isn't great, but okay for an example
    global client
    global register_definitions

    print('Resetting register data to default values ...')
    client.setup_registers(registers=register_definitions)
    print('Default values restored')

# commond slave register setup, to be used with the Master example above
register_definitions = {
    "COILS": {
        "RESET_REGISTER_DATA_COIL": {
            "register": 42,
            "len": 1,
            "val": 0
        },
        "EXAMPLE_COIL": {
            "register": 123,
            "len": 26,
            "val": [1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0]
        }
    },
    "HREGS": {
        "EXAMPLE_HREG": {
            "register": 93,
            "len": 9,
            "val": [29, 38, 0, 1600, 2150, 5067, 2564, 8450, 3456]
        }
    },
    "ISTS": {
        "EXAMPLE_ISTS": {
            "register": 67,
            "len": 1,
            "val": 0
        }
    },
    "IREGS": {
        "EXAMPLE_IREG": {
            "register": 10,
            "len": 1,
            "val": 60001
        }
    }
}

# alternatively the register definitions can also be loaded from a JSON file
# this is always done if Docker is used for testing purpose in order to keep
# the client registers in sync with the test registers
if IS_DOCKER_MICROPYTHON:
    with open('registers/example.json', 'r') as file:
        register_definitions = json.load(file)

# add callbacks for different Modbus functions
# each register can have a different callback
# coils and holding register support callbacks for set and get
register_definitions['COILS']['EXAMPLE_COIL']['on_set_cb'] = my_coil_set_cb
register_definitions['COILS']['EXAMPLE_COIL']['on_get_cb'] = my_coil_get_cb
register_definitions['HREGS']['EXAMPLE_HREG']['on_set_cb'] = \
    my_holding_register_set_cb
register_definitions['HREGS']['EXAMPLE_HREG']['on_get_cb'] = \
    my_holding_register_get_cb

# discrete inputs and input registers support only get callbacks as they can't
# be set externally
register_definitions['ISTS']['EXAMPLE_ISTS']['on_get_cb'] = \
    my_discrete_inputs_register_get_cb
register_definitions['IREGS']['EXAMPLE_IREG']['on_get_cb'] = \
    my_inputs_register_get_cb

# reset all registers back to their default value with a callback
register_definitions['COILS']['RESET_REGISTER_DATA_COIL']['on_set_cb'] = \
    reset_data_registers_cb

print('Setting up registers ...')
# use the defined values of each register type provided by register_definitions
client.setup_registers(registers=register_definitions)
# alternatively use dummy default values (True for bool regs, 999 otherwise)
# client.setup_registers(registers=register_definitions, use_default_vals=True)
print('Register setup done')

print('Serving as TCP client on {}:{}'.format(local_ip, tcp_port))

while True:
    try:
        result = client.process()
    except KeyboardInterrupt:
        print('KeyboardInterrupt, stopping TCP client...')
        break
    except Exception as e:
        print('Exception during execution: {}'.format(e))

print("Finished providing/accepting data as client")
brainelectronics commented 1 year ago

@GimmickNG as far as I can see from the automated tests umodbus.tcp._send_receive has some issues. May you can check this on your local machine with Docker?

brainelectronics commented 1 year ago

@beyonlo I've published https://test.pypi.org/project/micropython-modbus/2.4.0rc50.dev56/ for you, I initially wanted to wait for the tests to pass successfully, but I see the benefit in having the release out for testing

GimmickNG commented 1 year ago

@GimmickNG as far as I can see from the automated tests umodbus.tcp._send_receive has some issues. May you can check this on your local machine with Docker?

It should be fixed now - was because the examples did not call connect() on the client before running its methods. To maintain backwards compatibility, I made the synchronous TCP class auto-connect on initialize; since the asynchronous version cannot do that though, I've added a note in the constructor's documentation about the difference in behaviour.

beyonlo commented 1 year ago

@beyonlo I've published https://test.pypi.org/project/micropython-modbus/2.4.0rc57.dev56/ for testing. Initial tests passed after fixing the flake8 error locally

@brainelectronics Unfortunately I'm still trying to figure out how to use the register definitions, callbacks, etc on the async version. Coud you please or the @GimmickNG help me with an example for the ModBus TCP Slave?

On this reply I had talked about this subject but I think it was forgotten: " Sorry for my ignorance, but I have no idea how to integrate the register definitions, callbacks, etc to run with the async version. With the sync version there is the client.setup_registers(registers=register_definitions) that call the register_definitions, but how that is called by async version? Bellow there is as well a very simple sync example of tcp_client_example.py (based from the tcp_client_example.py ) that I trying to port to use with async version. Could you please, help me how to port that or point me a example of async version using the register_definitions?

My intention is start testing the ModBus TCP Slave and after ModBus RTU Slave, and finally the Masters RTU and TCP. All in async version :) "

Below is the ModBus TCP Slave example that I'm running without errors (got from the async examples: (async_examples.py), but there is no register definitions.

start_tcp_server.py: (async ModBus TCP Slave)

try:
    import uasyncio as asyncio
except ImportError:
    import asyncio

import os

print(f'MicroPython version: {os.uname()}')
from umodbus import version
print(f'Running ModBus version: {version.__version__}')

from umodbus.asynchronous.tcp import AsyncModbusTCP, AsyncTCP
from umodbus.asynchronous.serial import AsyncModbusRTU, AsyncSerial

async def start_tcp_server(host, port, backlog):
    server = AsyncModbusTCP()
    await server.bind(local_ip=host, local_port=port, max_connections=backlog)
    await server.serve_forever()

def run_tcp_test(host, port, backlog):
    asyncio.run(start_tcp_server(host, port, backlog))

run_tcp_test('192.168.43.143', 502, 2)

Output:

$ mpremote run start_tcp_server.py 
MicroPython version: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Running ModBus version: 2.4.0-rc57.dev56
GimmickNG commented 1 year ago

@beyonlo Sorry I didn't get back to you earlier, I forgot. Basically, the AsyncModbusTCP (i.e. AsyncTCP) class subclasses AsyncModbus which itself subclasses Modbus (like TCP/ModbusTCP), so whatever methods you would call on the client in the tcp_client_examples.py can also be called on the async examples as well.

In effect, you can replace ModbusTCP with AsyncModbusTCP and only change the processing code (i.e. call serve_forever() instead of while True: process())

I'll modify the examples directory later to add those examples - would probably benefit from refactoring the common bits out into a separate file.

By the way, @brainelectronics I think the files should be renamed if possible - the filename *_client_example.py can confuse people because it's actually a server, not a client according to the Modbus organization's redefinition:

The organization is using "client-server" to describe Modbus communications, characterized by communication between client device (s), which initiates communication and makes requests of server device(s), which process requests and return an appropriate response (or error message)

Likewise, the 'host' in *_host_example.py is actually a client, and not a server.

GimmickNG commented 1 year ago

@beyonlo I have updated the examples/ folder in my branch for the async TCP client/server ("host/client" in the repo for now). Although I haven't tested it yet (since the changes were made online), I think it should work since they share the same common code as the synchronous version. I'll do the same for the RTU version later.

brainelectronics commented 1 year ago

By the way, @brainelectronics I think the files should be renamed if possible - the filename *_client_example.py can confuse people because it's actually a server, not a client according to the Modbus organization's redefinition:

The organization is using "client-server" to describe Modbus communications, characterized by communication between client device (s), which initiates communication and makes requests of server device(s), which process requests and return an appropriate response (or error message)

Likewise, the 'host' in *_host_example.py is actually a client, and not a server.

@GimmickNG I'll do this together with other docs and example improvements after your PR is in

beyonlo commented 1 year ago

@GimmickNG Thanks to improve the examples :partying_face:

The async version stop on the await server.bind(). Details below:

To start the tests, I decide do use for now just the minimal example, without callbacks and so on.

$ cat async_tcp_slave_basic.py 

import uasyncio as asyncio

import os
print(f'MicroPython version: {os.uname()}')
from umodbus import version
print(f'Running ModBus version: {version.__version__}')

from rd import register_definitions

from umodbus.asynchronous.tcp import AsyncModbusTCP
async def start_tcp_server(host, port, backlog):
    server = AsyncModbusTCP()
    print('debug 1')
    await server.bind(local_ip=host, local_port=port, max_connections=backlog)
    print('debug 2')

    print('Setting up registers ...')
    server.setup_registers(registers=register_definitions)
    print('Register setup done')
    print('Serving as TCP client on {}:{}'.format(local_ip, tcp_port))
    await server.serve_forever()

# create and run task
task = start_tcp_server('192.168.43.143', 502, 2)
asyncio.run(task)

$ cat rd.py 

register_definitions = {
    "COILS": {
        "RESET_REGISTER_DATA_COIL": {
            "register": 42,
            "len": 1,
            "val": 1
        },
        "EXAMPLE_COIL": {
            "register": 123,
            "len": 26,
            "val": [1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0]
        }
    },
    "HREGS": {
        "EXAMPLE_HREG": {
            "register": 93,
            "len": 9,
            "val": [29, 38, 0, 1600, 2150, 5067, 2564, 8450, 3456]
        }
    },
    "ISTS": {
        "EXAMPLE_ISTS": {
            "register": 67,
            "len": 1,
            "val": 1
        }
    },
    "IREGS": {
        "EXAMPLE_IREG": {
            "register": 10,
            "len": 1,
            "val": 60001
        }
    }
}

Output:

$ mpremote run async_tcp_slave_basic.py 
MicroPython version: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Running ModBus version: 2.4.0-rc57.dev56
debug 1

Look that it stop on debug 1 print.

Below is the same example running in sync mode - works:

$ cat sync_tcp_slave_basic.py 

import os
print(f'MicroPython version: {os.uname()}')
from umodbus import version
print(f'Running ModBus version: {version.__version__}')

from rd import register_definitions

from umodbus.tcp import ModbusTCP

tcp_port = 502
local_ip = '192.168.43.143'

client = ModbusTCP()
is_bound = False
is_bound = client.get_bound_status()
if not is_bound:
    client.bind(local_ip=local_ip, local_port=tcp_port)

print('Setting up registers ...')
client.setup_registers(registers=register_definitions)
print('Register setup done')
print('Serving as TCP client on {}:{}'.format(local_ip, tcp_port))
while True:
    try:
        result = client.process()
    except KeyboardInterrupt:
        print('KeyboardInterrupt, stopping TCP client...')
        break
    except Exception as e:
        print('Exception during execution: {}'.format(e))
print("Finished providing/accepting data as client")

Output:

$ mpremote run sync_tcp_slave_basic.py 
MicroPython version: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Running ModBus version: 2.4.0-rc57.dev56
Setting up registers ...
Register setup done
Serving as TCP client on 192.168.43.143:502
GimmickNG commented 1 year ago

@beyonlo That was rather stupid of me - the bind method ends up calling wait_closed so it's effectively the same as serve_forever. Right, a workaround for the time being would be to remove line 165 and 167 of umodbus.asynchronous.tcp, but I'll have to do some more thorough cleaning later, because it seems like the serve_forever method doesn't actually lead anywhere (it's apparently not defined in any of the _itf classes)

beyonlo commented 1 year ago

@beyonlo That was rather stupid of me - the bind method ends up calling wait_closed so it's effectively the same as serve_forever. Right, a workaround for the time being would be to remove line 165 and 167 of umodbus.asynchronous.tcp

@GimmickNG I did that change as you told me, but another error happens. Bellow the details: async_tcp_changed

Output:

$ mpremote run async_tcp_slave_basic.py 
MicroPython version: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Running ModBus version: 2.4.0-rc57.dev56
debug 1
debug 2
Setting up registers ...
Register setup done
Serving as TCP client on 192.168.43.143:502
Traceback (most recent call last):
  File "<stdin>", line 26, in <module>
  File "uasyncio/core.py", line 1, in run
  File "uasyncio/core.py", line 1, in run_until_complete
  File "uasyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 21, in start_tcp_server
  File "umodbus/asynchronous/tcp.py", line 60, in serve_forever
AttributeError: 'AsyncTCPServer' object has no attribute 'serve_forever'

but I'll have to do some more thorough cleaning later, because it seems like the serve_forever method doesn't actually lead anywhere (it's apparently not defined in any of the _itf classes)

Don't worry, when you fix that, I will restart the tests!

GimmickNG commented 1 year ago

@beyonlo Please try the async tcp examples now, it seems to be working for the most part on my environment. However, I tested it with CPython and not Micropython, so there may be some differences in the implementation. Also, I get some struct errors in the async_tcp_client_example when using it with the async_tcp_host_example, but I'm not sure whether it's because of CPython or not. In any case, I'm not sure whether that's because of the underlying modbus implementation or the async tcp communications themselves - if you also get the same error, then I'll take a look at it probably on Thursday or Friday.

brainelectronics commented 1 year ago

@beyonlo I've published https://test.pypi.org/project/micropython-modbus/2.4.0rc62.dev56/ for testing.

@GimmickNG please check flake8 before pushing changes, builds are not even reaching the tests anymore

pip install -r requirements-test.txt
flake8 .
yamllint .

# tests can be locally checked by
docker build --tag micropython-test --file Dockerfile.tests .
docker compose up --build --exit-code-from micropython-host
docker compose -f docker-compose-tcp-test.yaml up --build --exit-code-from micropython-host
docker compose -f docker-compose-rtu.yaml up --build --exit-code-from micropython-host
docker compose -f docker-compose-rtu-test.yaml up --build --exit-code-from micropython-host
beyonlo commented 1 year ago

@beyonlo I've published https://test.pypi.org/project/micropython-modbus/2.4.0rc62.dev56/ for testing.

@brainelectronics Thank you for that!

@beyonlo Please try the async tcp examples now, it seems to be working for the most part on my environment. However, I tested it with CPython and not Micropython, so there may be some differences in the implementation.

@GimmickNG my words are: amazing job, amazing examples and thank you so much!

I did the preliminary tests of async ModBus TCP Slave (async_tcp_client_example.py) with the async ModBus TCP Master (async_tcp_host_example.py) and all passed.

My scenario is all with MicroPython 1.19.1: ESP32-S3 (Slave TCP) ------- WiFi AP ----- ESP32-S3 (Master TCP).

I tested as well with two Master TCP (one of them was the MODSCAN) working with the Slave TCP, and works like a charm with 2 connections. I still to not tested with more than two connections, but I want to.

I did as well a real test running together another asyncio application, the Microdot, and works perfect:

asyncio.run(slave_tcp_task)
await app.start_server(host='0.0.0.0', port=80, debug=False)

Also, I get some struct errors in the async_tcp_client_example when using it with the async_tcp_host_example, but I'm not sure whether it's because of CPython or not. In any case, I'm not sure whether that's because of the underlying modbus implementation or the async tcp communications themselves - if you also get the same error, then I'll take a look at it probably on Thursday or Friday.

That is strange, because I did exactly that test (but with MicroPython), and I have no errors.

Could you please to provide the example using the ModBus RTU (Slave and Master), like you did with TCP?

Ps: Could you think (maybe one more example) in a scenario (it is my case) where need to have both of Slaves (TCP and RTU) running simultaneously, where that both Slaves share the same register definitions and callbacks?

GimmickNG commented 1 year ago

@brainelectronics Sorry, I thought I'd fixed the flake8 errors in the IDE but evidently I missed a few. I've updated it now and as best as I can tell, there should be no more errors, although I don't have Docker at the moment so I can't run the tests themselves.

@beyonlo Wonderful, it's great to hear that! I think more connections should be supported, since my original version for Pycopy ran with some devices having around 5 or so connections. Granted, that was on the Ubuntu version which might be more powerful, but I don't see why it shouldn't work even with the microcontrollers. Also, I guess the errors must have just been because of CPython if you weren't getting them on Micropython.

I've updated the examples and added multi_client_example.py for your usecase. Both the TCP and RTU servers share the same set of register definitions, but unfortunately because of limitations with the callbacks at present, only one server can have the callbacks registered to it. In the future perhaps the callbacks could have the self argument passed to help identify the server which called the callback.

GimmickNG commented 1 year ago

Well, the build is no longer immediately failing...but I'm not exactly sure why the tests for the input registers are failing.

micropython-host    | Traceback (most recent call last):
micropython-host    |   File "mpy_unittest.py", line 373, in run_one
micropython-host    |   File "tests/test_tcp_example.py", line 595, in test_read_input_registers_single
micropython-host    |   File "mpy_unittest.py", line 97, in assertEqual
micropython-host    | AssertionError: (60001,) vs (expected) (60002,)
[...]
FAILED (failures=1, errors=0)
micropython-host    | DEBUG:tests.test_tcp_example:Initial status of COIL 150 length 19: [True, False, True, True, False, False, True, True, True, True, False, True, False, True, True, False, True, False, True], expectation: [True, False, True, True, False, False, True, True, True, True, False, True, False, True, True, False, True, False, True]
micropython-client  | Resetting register data to default values ...
micropython-host    | DEBUG:tests.test_tcp_example:Result of setting COIL 150 length 5 to [False, True, True, False, False]: True, expectation: True
micropython-client  | Default values restored
micropython-host    | DEBUG:tests.test_tcp_example:Status of COIL 150 length 5: [False, False, True, True, False], expectation: [True, False, True, False, True, True, False, False, True, True, False, True, False, True, True, False, True, False, True]

There's a +1 on line 580, so I guess it might be because I moved around the callback registrations, but I would've expected that to happen for the RTU example since the TCP example uses (effectively) the same set of callbacks as before. I'll take a look at it tomorrow.

beyonlo commented 1 year ago

Hello @GimmickNG

ModBus TCP (Slave and Master):

As you modified the examples, I run again the async ModBus TCP Slave (async_tcp_client_example.py) with the async ModBus TCP Master (async_tcp_host_example.py) but I found just a import error. Follow the diff:

$ diff examples_ORIG/async_tcp_client_example.py examples_CHANGED/async_tcp_client_example.py
24c24
< from examples.common.tcp_client_common import register_definitions
---
> from examples.common.register_definitions import register_definitions

With that change, all passed.

ModBus RTU (Slave):

I found errors on rtu_pins, so I fixed as far as I had my knowledge helped me. Follow the diff:

$ diff examples_ORIG/async_rtu_client_example.py examples_CHANGED/async_rtu_client_example.py 
28c28
< from examples.common.rtu_client_common import register_definitions
---
> from examples.common.register_definitions import register_definitions
30c30
< from examples.common.rtu_client_common import baudrate, uart_id, exit
---
> from examples.common.rtu_client_common import baudrate, uart_id
34d33
<                            rtu_pins,
35a35,36
>                            rtu_pins,
>                            ctrl_pin,
41d41
<                        pins=rtu_pins,
42a43,44
>                        pins=rtu_pins,
>                        ctrl_pin=ctrl_pin,
64,65c66
< task = start_rtu_server(addr=slave_addr,
<                         pins=rtu_pins,          # given as tuple (TX, RX)
---
> task = start_rtu_server(slave_addr=slave_addr,
66a68
>                         rtu_pins=rtu_pins,          # given as tuple (TX, RX)
70,71c72,74
<                         # ctrl_pin=12,          # optional, control DE/RE
<                         uart_id=uart_id)        # optional, default 1, see port specific docs
---
>                         ctrl_pin=15,            # optional, control DE/RE
>                         uart_id=uart_id         # optional, default 1, see port specific docs
>                         )

After that fixed, I had different error, that I don't know how to fix:

$ mpremote run async_rtu_client_example.py 
MicroPython infos: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Used micropthon-modbus version: 2.4.0-rc62.dev56
Using pins (17, 18) with UART ID 1
Traceback (most recent call last):
  File "<stdin>", line 75, in <module>
  File "uasyncio/core.py", line 1, in run
  File "uasyncio/core.py", line 1, in run_until_complete
  File "uasyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 41, in start_rtu_server
  File "umodbus/asynchronous/serial.py", line 53, in __init__
  File "umodbus/asynchronous/modbus.py", line 31, in __init__
AttributeError: 'AsyncRTUServer' object has no attribute 'set_params'

At least now the print pins works -> Using pins (17, 18) with UART ID 1

EDIT:

I've updated the examples and added multi_client_example.py for your usecase. Both the TCP and RTU servers share the same set of register definitions, but unfortunately because of limitations with the callbacks at present, only one server can have the callbacks registered to it.

I didn't try run the multi_client_example.py because the Slave RTU is not working yet, but I saw the code - very good!

In the future perhaps the callbacks could have the self argument passed to help identify the server which called the callback.

Do you mean that if the examples use classes (that has the self of course) instead just functions, is possible to share the callbacks between the Slave RTU and Slave TCP, like as the register_definitions?

GimmickNG commented 1 year ago

@beyonlo Thanks for correcting those errors, I didn't know the imports resulted in errors on Micropython, and I missed the syntax errors on the RTU examples.

AsyncRTUServer has no attribute set_params

Ah, I see. The whole deal around set_params and get_request is a bit strange and there's probably a much better way to implement it. Right now the async TCP interface is doing all the heavy lifting, whereas the async RTU server (not its interface) is doing the same task of reading data from the relevant sources. So right now the fix that works is to just define a blank set_params function in the RTU _itf (i.e. AsyncRTUServer) and it should work, but a real fix would be to call get_request() in Modbus.process, determine if the returned value is a Task (i.e. it is called as super().process() in AsyncModbus.process), and then have AsyncModbus.process perform the read, perhaps, instead of having AsyncTCPServer._accept_request doing it.

GimmickNG commented 1 year ago

As for why the tests fail, it seems to be because the response is probably sent before the values are incremented in the TCP client (i.e. server):

micropython-host    | DEBUG:tests.test_tcp_example:Status of IREG 10: (60001,), expectation: (60002,)
micropython-client  | Custom callback, called on getting IREGS at 10, currently: [60001]
micropython-client  | Incremented current value by +1 before sending response
micropython-host    | test_read_input_registers_single (TestTcpExample) ... FAIL
micropython-client  | Resetting register data to default values ...
micropython-client  | Default values restored

I'm not sure how the changes I made fit into this. I know that they cause this to happen, but the register definitions have (AFAIK) the same callbacks registered as before, but now it doesn't happen in order. Any idea @brainelectronics?

beyonlo commented 1 year ago

but a real fix would be to call get_request() in Modbus.process, determine if the returned value is a Task (i.e. it is called as super().process() in AsyncModbus.process), and then have AsyncModbus.process perform the read, perhaps, instead of having AsyncTCPServer._accept_request doing it.

@GimmickNG if you want first to do that real fix before the @brainelectronics generate another package and I restart the test, please feel free, I can wait for that! A real fix is a good idea :)

GimmickNG commented 1 year ago

Hmm...now that I think about it, I think it should be fine to let the read happen in the interface, since having the same read() code for both serial and TCP is unrealistic; the bigger issue would be the difference in the way that the synchronous and asynchronous versions handle processing. That is, in the synchronous versions, users are expected to call process() each time they want to process a request. Meanwhile, in the asynchronous versions, they can call process() if it's the serial server, but if it's TCP then it's not possible to call process() manually --- you need to call serve_forever().

I'm thinking of having the callback for open_connection() in the TCP server just be used to store references to the various stream readers and writers, so that when process() is called, the first reader that has data available will have its request processed - so that the user can call process() only when they want to process requests (instead of it happening asynchronously). However, it's possible that this could be a downgrade in terms of behaviour, since automatic processing might need to take preference over a consistent interface/behaviour.

I think ultimately, if people only use serve_forever() for the TCP server, then it should be possible to decide the behaviour "under-the-hood" later --- since serve_forever() can later be converted to calling await reader.read() on each of the available StreamReaders in an infinite loop. So for the time being, I think it can be left as is, even if it appears a bit convoluted.

beyonlo commented 1 year ago

Hello @GimmickNG

I readed many times your comments, but unfortunately I not understand the entire process of sync and async to give you a well formed opinion, because I can missing some understanding and can to talk some bullshit. But looking as end user of async lib (without think as are working under-the-hood), both async Slaves (TCP and RTU) (even I still can't success testing Slave RTU) shows very good in the examples, where both has a await client.serve_forever() and can easily integrate with others async applications. I see that you are using the asyncio.StreamWriter() and asyncio.StreamReader() for both (Slave TCP and RTU), so I understand that is possible to have the same behaviour to proccess on the TCP and RTU. But I think that I understand your point about async behaviour: that on the Slave TCP you do not have the control for each request/process, because it naturally happens automatically, different of Slave RTU. Well, maybe to add that callbacks on the Slave TCP is a good idea, because you will have the same behaviour for RTU and TCP, same code, still will be async (cooperative tasks with others async applications) and if you need some kind of better control about each stream, now we can, exactly as in the RTU already has.

Maybe @brainelectronics can give us a better comments about this subject!

GimmickNG commented 1 year ago

@beyonlo Yep, you've understood my post correctly. Either I try and make the RTU slave work the same way as the TCP slave (which I am not exactly sure of, but I have an idea) or I make the TCP slave work the same way as the RTU slave (which is easier but less ideal as it means the library user has to put in more effort to manage it).

GimmickNG commented 1 year ago

@beyonlo I've updated the branch; now the async RTU slave should behave similarly to the TCP slave, i.e. you can call bind() and it should process in the background, no need to call serve_forever(). While it'd be nice to make the async versions behave similarly to the sync versions, doing so would probably make development far more confusing, not to mention probably suboptimal in performance too.

beyonlo commented 1 year ago

@brainelectronics The @GimmickNG did changes on the async version. Could you please to generate a new version to me continue the tests? Or for now can I just to use the @GimmickNG branch https://github.com/GimmickNG/pycopy-modbus/tree/mpmodbus-compatibility/umodbus?

GimmickNG commented 1 year ago

@brainelectronics @beyonlo the builds are passing now, maybe it should be fine to create a new package

brainelectronics commented 1 year ago

@beyonlo I've published https://test.pypi.org/project/micropython-modbus/2.4.0rc78.dev56/ for testing, test are passing as @GimmickNG mentioned, I've just one last remark regarding unused __pycache__ files, then we can finally merge this 👍

beyonlo commented 1 year ago

@GimmickNG Hello!!

Today I restarted the tests with new version, but I have this error now on the Slave RTU:

$ mpremote run async_rtu_client_example.py 
MicroPython infos: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-994-ga4672149b on 2023-03-29', machine='ESP32S3 module with ESP32S3')
Used micropthon-modbus version: 2.4.0-rc78.dev56
Using pins (25, 26) with UART ID 1
Setting up registers ...
Register setup done
Traceback (most recent call last):
  File "<stdin>", line 76, in <module>
  File "uasyncio/core.py", line 1, in run
  File "uasyncio/core.py", line 1, in run_until_complete
  File "uasyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 64, in start_rtu_server
  File "umodbus/asynchronous/serial.py", line 64, in serve_forever
  File "umodbus/asynchronous/serial.py", line 119, in serve_forever
  File "uasyncio/core.py", line 1, in run_until_complete
  File "umodbus/asynchronous/serial.py", line 112, in _uart_bind
AttributeError: 'AsyncRTUServer' object has no attribute 'process'

I'm using your example async_rtu_client_example.py with a simple change: just uncomment ctrl_pin=15, # optional, control DE/RE because I use RS-485 with control pin. But running the original example, without this change, I have the same error.

EDIT: I tested as well in this version the Slave TCP using your example async_tcp_client_example.py and works fine!

GimmickNG commented 1 year ago

@beyonlo Thanks for notifying me. I'd confused the process() of AsyncModbusRTU and thought it belonged to AsyncRTUServer. Now it's more similar to how the TCP server interface works - AsyncModbusRTU's process() is passed as a callback to the AsyncRTUServer when it's created, and so now _uart_bind() should call req_handler instead -- hopefully it works now.

beyonlo commented 1 year ago

@brainelectronics Sorry to bore you again, but I think that is needed to generate another version to continue the tests because @GimmickNG changed the code right?

If you thing that is better I to do all tests in async version using the @GimmickNG repo (GimmickNG:mpmodbus-compatibility) while still have changes on the async code, please, let me know!

Thank you very much!

beyonlo commented 1 year ago

@beyonlo here you go with rc80

@brainelectronics the rc80 still have the same error:

$ mpremote run async_rtu_client_example.py 
MicroPython infos: (sysname='esp32', nodename='esp32', release='1.19.1', version='v1.19.1-966-g05bb26010 on 2023-03-13', machine='ESP32S3 module (spiram) with ESP32S3')
Used micropthon-modbus version: 2.4.0-rc80.dev56
Using pins (17, 18) with UART ID 1
Setting up registers ...
Register setup done
Traceback (most recent call last):
  File "<stdin>", line 76, in <module>
  File "uasyncio/core.py", line 1, in run
  File "uasyncio/core.py", line 1, in run_until_complete
  File "uasyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 64, in start_rtu_server
  File "umodbus/asynchronous/serial.py", line 64, in serve_forever
  File "umodbus/asynchronous/serial.py", line 119, in serve_forever
  File "uasyncio/core.py", line 1, in run_until_complete
  File "umodbus/asynchronous/serial.py", line 112, in _uart_bind
AttributeError: 'AsyncRTUServer' object has no attribute 'process'
brainelectronics commented 1 year ago

sorry @beyonlo, this was my bad, try the new rc81 instead, released just now https://test.pypi.org/project/micropython-modbus/2.4.0rc81.dev56/