pymodbus server does not work when using rfc2217

[nvincent-vossloh]

Versions

• Python: 3.10.12
• OS: Ubuntu 22.04.4 LTS
• Pymodbus: 03542f6b0843
• Modbus Hardware (if used): none

Pymodbus Specific

• Server: rtu - async : ./server_callback.py -c serial -f rtu --baudrate 115200 -p rfc2217://localhost:36000 -l debug
• Client: rtu - sync (not sure) : pymodbus.console serial --method rtu --port /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if00-port0 --baudrate 115200 Note: On the client side, I use version 3.5.4, otherwise the client complains about xonxoff not being a valid argument, but that is not the point of this issue.

Description

I want to do pymodbus on a remote serial interface accessible through rfc2217, however it looks like something is not working when the serial device is provided as an rfc2217 url, the server does not receive anything

In order to test this I have two rs485 interfaces looped back together with a cable:

• /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if00-port0
• /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if01-port0

When I use the server_callback.py on one of this interface (-p option) and pymodbus.console on the other, I can write and read registers from the pymodbus console.

However, if I setup ser2net to serve the serial interface used for the server and use the rfc2217 url to the server, then the client cannot communicate with the modbus server

Code and Logs

Serving serial interface through rfc2217 with ser2net:

config file:

connection: &conrs485
  accepter: telnet(rfc2217,mode=server),36000
  connector: serialdev(nouucplock=true),/dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if01-port0,115200e81
  options:
    max-connections: 10

rs485_ser2net.yaml.txt

Command line to start ser2net:

ser2net -d -d -c rs485_ser2net.yaml

Logs on server when using an rfc2217 url:

./server_callback.py -c serial -f rtu --baudrate 115200 -p  rfc2217://localhost:36000 -l debug
Run serial server
2024-09-06 17:39:46,973 DEBUG logging:103 Awaiting connections server_listener
2024-09-06 17:39:47,530 INFO  logging:97 Server listening.
Executing <Task pending name='Task-1' coro=<run_callback_server() running at /home/nicolas/work/siema/be/VOG/trial_pymodbus/pymodbus/examples/./server_callback.py:73> wait_for=<Future pending cb=[Task.task_wakeup()] created at /home/nicolas/.virtualenvs/pymodbus/lib/python3.10/site-packages/pymodbus/server/async_io.py:278> cb=[_run_until_complete_cb() at /usr/lib/python3.10/asyncio/base_events.py:184] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.558 seconds
2024-09-06 17:39:47,531 DEBUG logging:103 Connected to server

Error on client side when server use an rfc2217 url:

pymodbus.console serial --method rtu --port /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if00-port0 --baudrate 115200

----------------------------------------------------------------------------
__________          _____             .___  __________              .__
\______   \___.__. /     \   ____   __| _/  \______   \ ____ ______ |  |
 |     ___<   |  |/  \ /  \ /  _ \ / __ |    |       _// __ \\\____ \|  |
 |    |    \___  /    Y    (  <_> ) /_/ |    |    |   \  ___/|  |_> >  |__
 |____|    / ____\____|__  /\____/\____ | /\ |____|_  /\___  >   __/|____/
           \/            \/            \/ \/        \/     \/|__|
                                        v1.3.0 - 3.5.4
----------------------------------------------------------------------------

> client.write_register address=5 value=18 slave=6
{
    "original_function_code": "6 (0x6)",
    "error": "[Input/Output] Modbus Error: [Invalid Message] No response received, expected at least 4 bytes (0 received)"
}

patches

I had to patch pyserial in order to provide the fileno() operation:

diff --git a/serial/rfc2217.py b/serial/rfc2217.py
index 12636d4..463cee3 100644
--- a/serial/rfc2217.py
+++ b/serial/rfc2217.py
@@ -496,6 +496,9 @@ class Serial(SerialBase):
             self.close()
             raise

+    def fileno(self):
+        return self._socket
+
     def _reconfigure_port(self):
         """Set communication parameters on opened port."""
         if self._socket is None:

pyserial.patch.txt

and pymodbus to avoid providing write_timeout parameter to the Serial object created with serial_for_url():

diff --git a/build/.gitkeep b/build/.gitkeep
deleted file mode 100644
index e69de29b..00000000
diff --git a/pymodbus/transport/serialtransport.py b/pymodbus/transport/serialtransport.py
index 11759c2a..4edef6b7 100644
--- a/pymodbus/transport/serialtransport.py
+++ b/pymodbus/transport/serialtransport.py
@@ -33,7 +33,10 @@ class SerialTransport(asyncio.Transport):
         self.poll_task: asyncio.Task | None = None
         self._poll_wait_time = 0.0005
         self.sync_serial.timeout = 0
-        self.sync_serial.write_timeout = 0
+        if url.startswith("rfc2217://"):
+            self.sync_serial.write_timeout = None
+        else:
+            self.sync_serial.write_timeout = 0

     def setup(self) -> None:
         """Prepare to read/write."""

pymodbus.patch.txt

My thoughts

From what I understand of asyncio (very little), the logs

Executing <Task pending name='Task-1' coro=<run_callback_server() running at /home/nicolas/work/siema/be/VOG/trial_pymodbus/pymodbus/examples/./server_callback.py:73> wait_for=<Future pending cb=[Task.task_wakeup()] created at /home/nicolas/.virtualenvs/pymodbus/lib/python3.10/site-packages/pymodbus/server/async_io.py:278> cb=[_run_until_complete_cb() at /usr/lib/python3.10/asyncio/base_events.py:184] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.558 seconds

indicates that something is taking too long in one of the coroutine for which await is not used, unfortunately, despite spending time to locate the root cause I could not find anything meaningful.

If someone has any idea on where the issue might be or good pointers to debug this kind of asyncio issue I could eventually investigate further.

Thanks, Nicolas.

[janiversen]

First of all did you secure your loop back cabling adhere to the rs485 standard, you cannot "just" interconnect the 2 interfaces rs485 is a bit more complicated.

To me it looks like you do not define the same serial parameters for ser2net as for the usb, which might be the reason why the oymodbus server receives nothing.

The wait_for is used in clients together with the timeout parameter, but it should never trigger unless the client do not receive a response.

try testing with server_async.py that might provide a bit more information. It really seems as if the server never receives a request.

[janiversen]

Did you note you use different baudrates ! apart from that a lot of usb adapters have problems with that high a baudrate, and it's really not needed...try to test with e,g, 9.600

[nvincent-vossloh]

Thanks for your reply.

Please note that without rfc2217, it works perfectly fine with a cable between the two serial interfaces (I am using a StarTech 4 ports rs485 to usb):

./server_callback.py -c serial -f rtu --baudrate 115200 -p  /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if01-port0  -l debug
pymodbus.console serial --method rtu --port /dev/serial/by-id/usb-FTDI_USB__-__Serial_Converter_FT87SN9E-if00-port0 --baudrate 115200

I have amended the ser2net config file, as far as I understand, it should not matter as the script will reconfigure the serial interface as requested on the command line and/or default parameters.

The error/warning reported by asyncio occurs during the server initialization, the client is not involved at all in the error.

I initially gave the client commands in order to showcase the working environment: two rs485 interfaces connected to eachother directly without rfc2217 involved. As soon as the server side use an rfc2217 interface, the error is reported when the server starts. I apologize for the confusion.

I tried using server_async.py and I get the same results, it does not work when rfc2217 is involved:

./server_async.py -c serial -f rtu --port rfc2217://localhost:36000 --baudrate 115200 -l debug
2024-09-09 09:15:17,056 DEBUG logging:103 Awaiting connections server_listener
2024-09-09 09:15:17,612 INFO  logging:97 Server listening.
Executing <Task pending name='Task-1' coro=<async_helper() running at /home/nicolas/work/siema/be/VOG/trial_pymodbus/pymodbus/examples/./server_async.py:223> wait_for=<Future pending cb=[Task.task_wakeup()] created at /home/nicolas/.virtualenvs/pymodbus/lib/python3.10/site-packages/pymodbus/server/async_io.py:278> cb=[_run_until_complete_cb() at /usr/lib/python3.10/asyncio/base_events.py:184] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.558 seconds
2024-09-09 09:15:17,614 DEBUG logging:103 Connected to server

[janiversen]

A simple cable, might work depending on the devices connected to the cable....BUT a simple cable might cause heavy data loss if there are no pull-ups or its too short.

As I wrote earlier, did you try with a lower baud rate, 115200 is not supported by a lot of devices.

The task probably terminates because the asyncio server receives something bad from rfc2217. The task is terminated because "self.serving" which is a future() is set.

[nvincent-vossloh]

Tested at 9600 (ser2net config + server command line parameter), and I get the same behavior:

./server_async.py -c serial -f rtu --port rfc2217://localhost:36000 --baudrate 9600 -l debug
2024-09-09 11:02:53,893 DEBUG logging:103 Awaiting connections server_listener
2024-09-09 11:02:54,454 INFO  logging:97 Server listening.
Executing <Task pending name='Task-1' coro=<async_helper() running at /home/nicolas/work/siema/be/VOG/trial_pymodbus/pymodbus/examples/./server_async.py:223> wait_for=<Future pending cb=[Task.task_wakeup()] created at /home/nicolas/.virtualenvs/pymodbus/lib/python3.10/site-packages/pymodbus/server/async_io.py:278> cb=[_run_until_complete_cb() at /usr/lib/python3.10/asyncio/base_events.py:184] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.562 seconds
2024-09-09 11:02:54,454 DEBUG logging:103 Connected to server

What exactly do you mean by

The task probably terminates because the asyncio server receives something bad from rfc2217

When the log appears, the client has not been started yet.

[janiversen]

Pymodbus uses the same transport class for both server and client, so it one works the other works too, from a software POW.

The parameters are internally handled by the transport class, but server/client can have different defaults.

The transport layer communicates with the RFC2217 long before the client is started, otherwise it would not be able to start listening. What can happen is that the RFC2217 receives a setting it does not like and closes the connection or for other reasons closes the connection to pymodbus.

I also not that you do not open the port in the same way for the server rfc2217://localhost:36000 and the client /dev/serial/by-id/usb-FTDI_USB-Serial_Converte... which means you cannot claim the client works.

I suspect, but do not know, that you actually do not use the usb, but instead some sort of tcpip (indicated by localhost).

[janiversen]

You can try rfc2217://localhost:36000 with wget since it is a http request, to check if there are permission problems, or "rfc2217:" is an unknown http protocol

Actually I wonder how this is ever going to work, you listen on a socket but the client uses the rfc2217 directly.

[nvincent-vossloh]

Please note that during my test, I test only one side in rfc2217, that is why the port parameter in the command lines differs (rfc2217://localhost:36000 vs /dev/serial/by-id/...)

Here is what I have tested so far:

(I have replaced long tty file devices names with their shorter logical names for better readability)

I am quite surprised that setup C works. For future reference, here are the commands involved:

Setup A:

# server
./server_async.py -c serial -f rtu --port /dev/ttyUSB1 --baudrate 9600 -l debug
# client
pymodbus.console serial --method rtu --port /dev/ttyUSB2 --baudrate 9600

Also tested at 115200 by replacing 9600 in both commands above.

Setup B

rs485_ser2net.yaml (serial interface configured at 9600 bauds to localhost:36000)

connection: &conrs485
  accepter: telnet(rfc2217,mode=server),36000
  connector: serialdev(nouucplock=true),/dev/ttyUSB2,9600
  options:
    max-connections: 10

# ser2net:
ser2net -d -d -c rs485_ser2net.yaml
# server 
./server_async.py -c serial -f rtu --port rfc2217://localhost:36000 --baudrate 9600 -l debug

The logs on the server side reports the error as soon as it is started. If I try to run the client with the same command as setup A, commands sent in the client console will fail:

> client.read_holding_registers address=1 count=10
{
    "original_function_code": "3 (0x3)",
    "error": "[Input/Output] Modbus Error: [Invalid Message] No response received, expected at least 4 bytes (0 received)"
}

Setup C

I tested this setup only after your last message. I am really surprised to see this one working. The ser2net config file is identical to setup B. Here are the commands used:

# ser2net:
ser2net -d -d -c rs485_ser2net.yaml
# server
./server_async.py -c serial -f rtu --port /dev/ttyUSB1 --baudrate 9600 -l debug
# client
pymodbus.console serial --method rtu --port rfc2217://localhost:36000 --baudrate 9600

I tried to change the baudrate on server and client command line to 115200 and 57600 and leaving the ser2net config at 9600, and it still works.

I suspect that the pymodbus server might be sending an extra configuration not supported by the rfc2217 transport layer or by ser2net.

You can try rfc2217://localhost:36000 with wget since it is a http request, to check if there are permission problems, or "rfc2217:" is an unknown http protocol

I am not sure I understand this, rfc2217 is not an http protocol. Anyway I tried with wget and I obtain this:

wget rfc2217://localhost:36000
rfc2217://localhost:36000: Unsupported scheme ‘rfc2217’.

Thanks again for your patience.

[janiversen]

Since setup A works, there are no problem with the physical devices.

Setup C, contains an error a client do not listen, but only does connect.

Setup B, is asking the impossible, a server cannot connect to a tcp/ip port !

The pymodbus configuration is wrong, because you ask pymodbus to listen on tcp/ip port 3600, but that is already done by ser2net. Remember a server listen and do not connect. You have configured a http (NOT html) which means the server tries to do a tcp/ip listen on that. A server cannot connect to a socket, but only listen on a socket, alternatively it can open a device.

Pymodbus server is not sending anything extra, since it is the same transport class as the client, however the difference is that the client starts by connection, while the server just opens the port and listen for a connection.

[nvincent-vossloh]

Since setup A works, there are no problem with the physical devices.

Agree.

Setup C, contains an error a client do not listen, but only does connect.

In setup B and C, ser2net process is a server which listen on a socket (localhost:36000) and will forward every bytes received on the serial interface (/dev/ttyUSB2) to the client connected to localhost:36000 (if a client has connected to it) and every bytes received on the socket is written on the serial interface. In setup C, there is a connection from the pymodbus.client to the ser2net at the rfc2217 level (telnet).

Setup B, is asking the impossible, a server cannot connect to a tcp/ip port !

I understand your point, but in Setup B, the connect happens only from server_async.py to ser2net (which is a server) and that connect is handled by pyserial and done with rfc2217 protocol (telnet).

The pymodbus configuration is wrong, because you ask pymodbus to listen on tcp/ip port 3600, but that is already done by ser2net.

No, I do not ask pymodbus (client or server) to listen on 36000, I provide them the url of the rfc2217 serial interface for which pyserial/rfc2217.py will connect to (localhost:36000).

Remember a server listen and do not connect.

Agree.

You have configured a http (NOT html) which means the server tries to do a tcp/ip listen on that.

I have configured ser2net to open a device and listen on port 36000, once ser2net is started it can accepts connection made by telnet localhost 36000 or using pyserial with the proper url (rfc2217://localhost:36000). There is no http nor html involved.

A server cannot connect to a socket, but only listen on a socket, alternatively it can open a device.

Agree.

Pymodbus server is not sending anything extra, since it is the same transport class as the client, however the difference is that the client starts by connection, while the server just opens the port and listen for a connection.

I do not completely agree. In setup B and C, the instance (server or client) going through ser2net connects to localhost:36000, however the connect is not done at pymodbus level, but at pyserial level (see rfc2217.py above). Please note that the -c/--comm flag passed to server_async.py is serial, and not tcp or udp.

I hope this clarify my setup and the issue I am facing. Thanks.

[janiversen]

Well as you put it, this is clearly a problem outside pymodbus, and might be in pyserial, so I suggest you open an issue there. It seems it does not like the url you provide or have some other problem.

And just to be sure, pymodbus server only opens the device (using pyserial) and sends absolutely nothing. Pymodbus client on the other hand opens the device and sends a request. In both cases (comm == serial) the transport code passed is the same.

There are 3 parts which can have a problem:

• pyserial
• python asyncio
• pymodbus

Write a small program that opens the localhost:36000 with pyserial, and then let it wait in a read for a couple of minutes.

Add asyncio create_server and repeat.

If those 2 works, then there is a problem in pymodbus (which I cannot pinpoint), otherwise it is out of our hands.

[nvincent-vossloh]

Thanks for your input.

Well as you put it, this is clearly a problem outside pymodbus, and might be in pyserial, so I suggest you open an issue there. It seems it does not like the url you provide or have some other problem.

Yes, I will try to create a testcase for them to work on.

And just to be sure, pymodbus server only opens the device (using pyserial) and sends absolutely nothing.

I agree, the pymodbus sends nothing, however it configures the serial interface from the configuration provided on the command line, I thought it could be the configuration of the serial interface, but since setup C (pymodbus client using an rfc2217 serial interface) is working. It might be something else.

Pymodbus client on the other hand opens the device and sends a request. In both cases (comm == serial) the transport code passed is the same.

Ok, thank you for the clarification.

There are 3 parts which can have a problem:

• pyserial
• python asyncio
• pymodbus

Write a small program that opens the localhost:36000 with pyserial, and then let it wait in a read for a couple of minutes.

I did that, it is working fine: ./test_pyserial_read.py rfc2217://localhost:36000 --baudrate 9600 test_pyserial_read.py:

#!/usr/bin/env python

import serial
import asyncio
import argparse

def read_serial(port, baudrate):
    ser = serial.serial_for_url(port, baudrate=baudrate)
    while True:
        print(ser.read(1))

def main():
    parser = argparse.ArgumentParser(description='Read data from serial port')
    parser.add_argument('port', help='Serial port')
    parser.add_argument('--baudrate', type=int, help='Baudrate', default=115200)

    args = parser.parse_args()
    read_serial(args.port, args.baudrate)

if __name__ == '__main__':
    main()

Add asyncio create_server and repeat.

I do not understand the role of create_server and do not know what parameters I should pass. I tried adding async keyword and run through asyncio.run(...) and it still works properly (receiving bytes):

#!/usr/bin/env python

import serial
import asyncio
import argparse

async def read_serial(port, baudrate):
    ser = serial.serial_for_url(port, baudrate=baudrate)
    while True:
        print(ser.read(1))

def main():
    parser = argparse.ArgumentParser(description='Read data from serial port')
    parser.add_argument('port', help='Serial port')
    parser.add_argument('--baudrate', type=int, help='Baudrate', default=115200)

    args = parser.parse_args()
    asyncio.run(read_serial(args.port, args.baudrate))

if __name__ == '__main__':
    main()

Thanks again for your time.

[janiversen]

Closing since this is primarily a pyserial problem (see patches needed). If pyserial decides to make a new release we are happy to use it.