Support for multiple units per connection

[git-developer]

First of all: thanks for this project. It's really useful for me!

I successfully use modbus-mqtt in a project to read a single energy meter via TCP. This works fine.

I'm now evaluating whether modbus-mqtt can be used for a different project where multiple energy meters have to be read via RTU.

Current setup

I managed to read a single energy meter via RTU using the following configuration:

{
  "proto": "rtu",
  "tty": "/dev/ttyUSB0",
  "baud_rate": 19200,

  "unit": 30,
  "registers": [
    { "name": "firmware", "address":   1, "register_type": "holding", "scale": -1, "interval": "100y" },
    { "name": "energy",   "address":  28, "register_type": "holding", "type": "u32", "scale": -2, "interval": "30s" },
    { "name": "power",    "address":  38, "register_type": "holding", "scale": -2, "interval": "30s" }
  ]
}

This works fine, too. Now I'd like to add a second energy meter that is connected to the same bus. The second meter has the same model as the first one, only the unit (slave id) differs. My project includes 15 meters in total.

When I read the documentation right, modbus-mqtt supports a 1:1 relation between connection and unit. Is that correct? If so, the serial connection would have to be declared for each energy meter. I'm concerned about problems when requests for multiple units are written to the serial device without coordination.

Suggestion: Multiple units per connection

Would it be possible to allow a 1:n relation between connection and units? For example:

{
  "proto": "rtu",
  "tty": "/dev/ttyUSB0",
  "baud_rate": 19200,

  "units": [
    {
      "unit": 30,
      "registers": [
        { "name": "firmware", "address":   1, "register_type": "holding", "scale": -1, "interval": "100y" },
        { "name": "energy",   "address":  28, "register_type": "holding", "type": "u32", "scale": -2, "interval": "30s" },
        { "name": "power",    "address":  38, "register_type": "holding", "scale": -2, "interval": "30s" }
      ]
    }
    {
      "unit": 31,
      "registers": [
        { "name": "firmware", "address":   1, "register_type": "holding", "scale": -1, "interval": "100y" },
        { "name": "energy",   "address":  28, "register_type": "holding", "type": "u32", "scale": -2, "interval": "30s" },
        { "name": "power",    "address":  38, "register_type": "holding", "scale": -2, "interval": "30s" }
      ]
    }
  ]
}

With such a feature I'd be able to implement my project with modbus-mqtt. Without it, I'll probably need to look for an alternative, which I'd like to avoid.

Optional feature: Unit templates

When multiple counters of the same model are connected, their declaration would introduce redundancy. This could be avoided by a template mechanism, e.g.:

{
  "proto": "rtu",
  "tty": "/dev/ttyUSB0",
  "baud_rate": 19200,

  "unit_templates": {
    "foo": {
      "address_offset": -1,
      "register_type": "holding",
      "registers": [
        { "name": "firmware",  "address":   1, "scale": -1, "interval": "100y" }
      ]
    }
  },

  "units": [
    { "unit": 30, "template": "foo" },
    { "unit": 31, "template": "foo" }
  ]
}

Such a the template mechanism would be nice to have, but is not required to complete my project.

Unfortunately, I have no experience in Rust so I can't supply a pull request. When there's a chance to implement this feature, I'm ready to help.

[bjeanes]

Hi @git-developer

Thanks for the excellent issue. Speaking generally about your first proposal: I think it's a good idea.

When I read the documentation right, modbus-mqtt supports a 1:1 relation between connection and unit. Is that correct?

That's right. However, this is because the Modbus library I am using internally does this: https://github.com/slowtec/tokio-modbus/blob/5384621ebe7465f64be2dce399554a36dbc54905/src/client/rtu.rs#L12-L28.

I am not opposed to changing the config payload to support representing multiple units on the same connection, but I am not sure if doing so would guarantee that the underlying library is coordinating the communications appropriately. This isn't such a concern with TCP, but with the RTU/serial, I can see why it would be a concern.

I might be able to mitigate it by having a kind of lock around RTU connections, but I am not sure without a way to test this. I might be a bit out of my depth on it for a while too.

Unfortunately, the only Modbus device I have is a TCP one (my Sungrow inverter) and this issue is actually the first confirmation for me that the RTU connection works at all.

I may need to research and experiment a bit with the Modbus library and read through it's serial code to make an estimate about whether this is possible.

If connections were to become multi-unit, then I think the topic scheme for register values would have to change. One might poll the same register number on two units and provide no name, which right now would conflict. I need to think about this a bit.

Practically speaking, this will be a hard one for me to iterate on without any Serial modbus devices, let alone a multi-unit one.

I am not going to have much free time for work in the next few weeks, but I am interested in helping you use this in your project. Perhaps in a couple of weekends I can come up with a branch for you to test, but I won't know how much time I'll have until I have the free time 🥲.

---

On your second proposal, this is probably best to be another issue so that we can discuss each individually. Broadly speaking, I think the problem you are trying to solve merits discussion, but I don't think there's enough usage of how common patterns would vary in the wild. The template solution you propose is an interesting idea but perhaps premature.

I think I'd prefer to let this issue simmer a bit and hopefully get a few more users using the software to see what kind of a solution is a best fit. In the interim, I suggest writing a little script which generates your JSON config. That'll make it easy to de-dupe on your end without having to fix it here yet.

[git-developer]

Thanks for understanding my issue and for your quick and detailed response.

I see that is makes no sense to implement this challenging feature when you don't even own an RTU device. Please don't put efforts into this topic unless you or someone else can benefit from it. After writing this issue, I managed to find alternative applications (spicierModbus2mqtt and mqmgateway) that both cover my use case. I'd prefer the tech stack and architecture of your project, but that's no sufficient justification for the implementation effort required.

I agree that my second suggestion is different and thus should be moved to in a different issue (or discussion, but this GitHub feature is not enabled here). It's just an idea and should be evolved before implementation. I'm fine with waiting for other users before taking any further steps, even if that means that it will not be implemented at all.

I'm aware that your code is mainly a library written for your specific use-case of integrating a Sungrow WiNet-S device. Nevertheless I have the impression that it is of high quality, clean, concise and is perfectly re-usable for other scenarios (such as my first project). All the other similar projects that I found until now (currently 8, and counting...) are either less mature or harder to understand & configure. Well done!

From my point of view, this issue may be closed because there's no succeeding step (except waiting for comments or spare time). I'll leave it up to you to keep it open to be visible or to close it.

[bjeanes]

I'm aware that your code is mainly a library written for your specific use-case of integrating a Sungrow WiNet-S device.

That was certainly the driving use-case for me, but I very much intended this to be a generic component in others' stacks.

or discussion, but this GitHub feature is not enabled here

I've just enabled it.

All the other similar projects that I found until now (currently 8, and counting...) are either less mature or harder to understand & configure. Well done!

Yup that's why I started this, as I first searched too.

Nevertheless I have the impression that it is of high quality, clean, concise and is perfectly re-usable for other scenarios

That is my intent, but I am also a bit of a Rust noob so the "high-quality" thing is probably debatable. It's certainly the direction I intend it to go!

I'll leave it up to you to keep it open to be visible or to close it.

I'll leave it open. I want to chew on it a bit and see if I can answer the question as to whether interacting with a shared Serial device is possible. Thankfully, in Rust it should be the case that if the type system allows me to do it, it's probably safe to do so.

[bjeanes]

After writing this issue, I managed to find alternative applications (spicierModbus2mqtt and mqmgateway) that both cover my use case. I'd prefer the tech stack and architecture of your project, but that's no sufficient justification for the implementation effort required.

Hey @git-developer, how did you end up fairing with either of these other projects? I'm interested to hear your experience and comparison.

It occurred to me earlier this week that perhaps you can get me enough info to answer the question I had earlier about how RTU connections on same port would behave:

I am not opposed to changing the config payload to support representing multiple units on the same connection, but I am not sure if doing so would guarantee that the underlying library is coordinating the communications appropriately. This isn't such a concern with TCP, but with the RTU/serial, I can see why it would be a concern.

I might be able to mitigate it by having a kind of lock around RTU connections, but I am not sure without a way to test this. I might be a bit out of my depth on it for a while too.

Basically, if you defined the two connections to the two units separately under the current config and set them to read registers from each unit with a relatively low interval (so rapid reads), you might be able to tell me if it crashes or does anything funky.

If it doesn't, then the underlying library might already handle this use-case internally by re-using the same handle to the serial connection. If it works but then does occasionally crash or get corrupted results, it tells me I need to add some kind of locking around this (or implement my own Modbus implementation). OTOH if one of them simply fails to connect, that tells me the underlying library just can't talk to multiple units over the same RTU connection.

What do you think?

[git-developer]

I'm interested to hear your experience and comparison.

In the end, I decided to use mqmateway for my project. The configuration contains a section for (multiple) modbus networks containing the connection parameters, and a separate section for Modbus/MQTT-Mappings. The unit (slave id) is part of the mapping which allows using multiple devices. The configuration contains options to control how to poll the Modbus network (refresh for the frequency and poll_group for batch behavior). Having a batch polling mechanism in place means that Modbus read and write operations are managed and synchronized by the software.

Example configuration

```yaml modmqttd: converter_search_path: - /usr/lib/modmqttd converter_plugins: - stdconv.so modbus: networks: - name: net-1 device: /dev/ttyUSB0 baud: 19200 parity: E data_bit: 8 stop_bit: 1 poll_groups: - { register: 30.1, count: 20 } - { register: 31.1, count: 20 } mqtt: client_id: mqmgateway refresh: 20s broker: host: broker-host objects: - topic: mqmgateway/net-1/meter-30/firmware state: { register: net-1.30.1, converter: "std.divide(10,1)" } refresh: 86400s # ⋮ (additional declarations) - topic: mqmgateway/net-1/meter-31/registers state: { register: net-1.31.2 } ```

I contributed a few PRs adding required type converters for my hardware and a Docker based CI build. After that, my use case was covered. Actually, I'm using mqmgateway for the multi-device RTU project, but I keep using modbus-mqtt for my single-device TCP project - because it works and there was no need to touch it.

What do you think?

It is possible to find out experimentally what actually happens, but you can't derive if the observed behavior (works/crashes) is intended or by accident. If you'd like to get insights about the features/behavior of the modbus library, its documentation and/or code should be a better reference IMHO, at least as a starting point. I had to learn that different Modbus devices behave differently, so you can't predict how other Modbus devices behave based on tests with a single device.

My projects are completed, I'm happy now and don't need further efforts here. If you're curious and motivated to extend modbus-mqtt, I think the best would be to inspect the underlying library and make some tests with real RTU devices.

Last but not least thanks for your motivation and for maintaining this project!

[bjeanes]

In the end, I decided to use mqmateway for my project.

Neat. I had a look at both options and that's definitely what I would have chosen too. It looks quite solid!

It is possible to find out experimentally what actually happens, but you can't derive if the observed behavior (works/crashes) is intended or by accident.

Of course. My hope was to essentially "fuzz" the concurrency handling here. With a fast enough interval and running for long enough, that would be a good indication that the library was designed to be resilient in this case.

If you'd like to get insights about the features/behavior of the modbus library, its documentation and/or code should be a better reference IMHO, at least as a starting point.

Yes agreed. I did have a quick look into the code around this when you first opened this issue. I'll do so again if I pursue this regardless of your help. At the time, it didn't seem that it was directly handled, but I didn't dig into the async runtime itself, which might have provided its own guarantees.

I had to learn that different Modbus devices behave differently, so you can't predict how other Modbus devices behave based on tests with a single device.

Of course, though I think in this case the goal is to determine through experimentation if reads/writes to the serial device are synchronised, which would be immaterial to the Modbus device in question, though I agree that an unstable Modbus device could confound the interpretation of that experiment.

My projects are completed, I'm happy now and don't need further efforts here.

Yep, fair enough! I expected you would have solved the issues on your end, but thought I'd ask just in case you had some free time, since I don't have any RTU Modbus devices handy and had hoped to hack on this on the weekend while at a conference. I don't know why it occur to me months ago that merely having you define the two connections separately and report your findings would be sufficient insight into whether these could realistically be defined.

If you're curious and motivated to extend modbus-mqtt, I think the best would be to inspect the underlying library and make some tests with real RTU devices.

I'll just have to acquire some Modbus RTU devices or test harness. Looking at the code is already on the agenda, but I had already done so and couldn't find any evidence of synchronisation.

I fear the most likely outcome is that I need to implement my own Modbus framing, but that is probably beyond my abilities and/or time availability 😞.

Anyway, thanks for writing back!