Add MQTT support to publish readings

[MarsWarrior]

My main Domotica backbone is build upon MQTT. Other sensors and my main system (openHAB) communicate over MQTT. I also use MQTTWarn from Jan-Piet Mens, which enables me to connect to several other services.

It would be very nice if the dsmr-reader could be extended with MQTT support:

• A page that would configure the MQTT broker IP and ports
• A page for configuring DSMR to MQTT topics:
• -- Per DSMR item a separate entry (or more than one entry) with the designated MQTT topic
• -- If an entry is empty, the topic is not used
• An option to send the topic as plain text (default) or formatted as JSON
• An option to defer updating a topic if topic is not changed. A time-out could be used to force the update every 5 minutes for instance. Default electricity is updated every 10seconds, and gas every hour (at least in DSMR 4.x).

Gasmeter example:

0-1:96.1.0(xxxx)                 -> /p1/gas/id xxxx
0-1:24.2.1(yyyy)(zzz.zzz*m3)     -> /p1/gas/timestamp_value yyyy
                                 -> /p1/gas/value zzz.zzz
0-1:24.4.0(1)                    -> /p1/gas/valve_position 1

As you can see, the 24.2.1 gasmeter value requires two topics in this case using raw values.

[dennissiemensma]

Hi, thank you for your request. Although you specified quite some details, I currently lack MQTT experience with the setup you are trying to describe here.

Do you simply wish to use dsmr-reader for only reading your meter and have it posted to another tool using MQTT?

If so, then this might be an excellent addition to the #96 feature (stand alone datalogger client). In #140 I've added an API for posting such data to dsmr-reader using a bare bone client. A small datalogger client will suffice either way. Using the whole dsmr-reader application for it is overkill anyway.

Besides, currently the application's datalogger is tightly coupled to the database. You can either extract the datalogger code from dsmr-reader/dsmr_datalogger/services.py or have the dsmr_datalogger process (in supervisor) push all data to a sqlite database, which is unsupported but will work anyway for just logging readings. You can then write a small script to export that data using MQTT. Or it could be a feature for #96.

Please clarify so I have a better understanding of your request.

[MarsWarrior]

TL;DR Yes, please add MQTT support to issue #96.

Do you simply wish to use dsmr-reader for only reading your meter and have it posted to another tool using MQTT?

Hmmm, difficult question. The dsmr-reader creates nice dashboards, but on the other hand I do prefer to have my dashboards elsewhere.

I didn't notice the #96 feature. That could indeed be a good solution for the MQTT support. A simple configuration file could then be used for instance to map the DSMR codes to MQTT topics, backed by a nice webinterface if possible.

My solution then would be:

• Standalone DSMR datalogger connected through MQTT, running on the RPi already hanging around near the smart meter. Payload is formatted as JSON and/or raw
• MQTTWarn with the InfluxDb plugin
• Grafana to display the time-series from the InfluxDb
• Other MQTT clients such as sensors and of course OpenHAB

I don't have any experience with the python library for MQTT, but the examples seem simple and straightforward to connect & publish MQTT topics.

import paho.mqtt as mqtt
import time

mqttc=mqtt.Client("dsmr-reader")
mqttc.connect("127.0.0.1" 1883, 60, True)
mqttc.loop_start()
while True:
    # do all the dsmr stuff every 10 seconds
    mqttc.publish("dsmr/x/y/z","{value: xyz}")
    time.sleep(10)# sleep for 10 seconds

[dennissiemensma]

Thank you for your input. Due to holiday I was unable to respond earlier. I will try to complete #96 for the current release, but I can't promise anything yet.

[dennissiemensma]

@MarsWarrior it has been quite some while, but I'm currently active again to perform some maintenance for this project.

Is your request here still pending or did you find an alternative in the meantime?

[MarsWarrior]

@dennissiemensma , yes my request is still pending. I'm not using any alternative.

But as you will understand: I'm in no hurry obviously. But if you want to add this feature now, please do!

[dennissiemensma]

I might give it a try. Therefor I'll need to know the exact setup you are using so I can reproduce and test it. I don't need to know every detail about the system, just the entrypoint for the MQTT and how I can see whether the readings are successful. Can you provide some info?

[dennissiemensma]

@MarsWarrior see my comment above. Do you have more information about how I can reproduce your setup at home? Because it' really hard to create something that I cannot test or verify that it works.

[thomasvnl]

I just saw that I read the title of this issue differently than what I mean, therfore this small addition:

I want to suggest to @MarsWarrior that adding publishing of MQTT messages from this dsmr-reader package feels imho outside of the scope of what this package is trying to do: logging results (but please, @dennissiemensma , correct me if I'm wrong).

You could write a tiny process with the paho.mqtt client that reads data from the serial port on your raspberry pi (see dsmr_datalogger/services.py) and publishes to whatever topic you want for further processing. That way you don't need anything from the dsmr-reader package which saves you installing dependencies and such.

My addition below could be added to a new feature request for listening on a certain topic to read in MQTT payloads from an external source.

-- New Feature --

@dennissiemensma I have a setup in which I send these kind of MQTT payloads to a topic like _home/power/dsmr/DEVICEID:

{
  "timestamp" : "170107114403W",
  "readings" : {
    "energy_delivered_tariff1" : "1414657",
    "energy_delivered_tariff2" : "1513958",
    "energy_returned_tariff1" : "331553",
    "energy_returned_tariff2" : "797366",
    "power_delivered" : "318",
    "power_returned" : "0",
    "gas_delivered" : "1702613",
    "gas_timestamp" : "170107110000W"
  }
}

MQTT requires a broker service (it's a socket with some extra stuff) like mosquitto, which runs on raspberry pi's too.

These messages come from a standalone ESP8266 with P1 Port Shield I designed, but you can fake these messages as well with a little routine that spews them out on the MQTT Broker every 10 seconds.

I was trying to create something like you have build with your dsmr-reader packge but then with MQTT + NodeJS + MongoDB but it fails miserably on the NodeJS side of things (NodeJS being Async, sometimes not so obvious). That is when I Googled and I came across this package. I have experience with Python + MQTT so I might consider adding stuff to your dsmr-reader instead to support MQTT (cannot say how long it would take), to work with your dashboarding (cause I love it!). I would appreciate some help with the Django stuff, as my Django Kung-Fu isn't as advanced as I would like it to be 😛

[dennissiemensma]

Hi @thomasvnl thank you for your well detailed comment and request.

I'm not sure whether I understand everything corectly, but I won't integrate any MQTT features. I do however try to help people, whenever possible, to have them run their own scripts locally. For example #187.

That's also the only reason this ticket is still open, because I closed another ticket regarding a stand-alone datalogger.

Some information about the scope of the project regarding data input:

• The builtin datalogger is only intended for the project's original target (dedicated application on RaspberryPi).
• There is however support to insert telegrams (readings) via the API, to allow people to run this project whereever they'd like.
• The API should be the only (other) way to input data.

And data output:

• I do support features for exporting data, such as CSV, MinderGas and PVOutput (todo, #9).
• Exporting readings (the very source data of the project) to MQTT seem a bit overkill and in a grey area regarding scope.

If I do read your request correctly and you do want dsmrreader to export readings to MQTT, then I think it would be better to use a more simple parser, such as this one.

• It requires a whole lot less dependencies than this project.
• It supports CRC-checking in the parser, which is nice.
• It's also used by the home-assistant project and should be easy to use (as you'd only need 8 values in your example).
• If you like to use DSMR-reader as well, you could have your Python script both send the telegram to DSMR. And have the dsmr_parser extract the data you're looking for (and send them to your MQTT setup).

I hope this helps you a lot in finding a suitable solution. I'll try to assist when possible.

[thomasvnl]

Hej @dennissiemensma,

Nice detailed response regarding the scope of the project. I don't think what I meant has reached you entirely so I will try to clarify what I meant in my previous response, because I was also replying to MarsWarrior's query.

But first:

Scopewise

• The builtin datalogger is only intended for the project's original target (dedicated application on RaspberryPi). I fully agree that this is the way it should be!

• There is however support to insert telegrams (readings) via the API, to allow people to run this project whereever they'd like. Really nice that it is available.

• The API should be the only (other) way to input data. This is where my earlier message becomes relevant, will clarify below.

Data output

• I do support features for exporting data, such as CSV, MinderGas and PVOutput (todo, #9). Saw it, really nice.

• Exporting readings (the very source data of the project) to MQTT seem a bit overkill and in a grey area regarding scope. Exporting to MQTT was certainly not what I meant 😄. This is more in line with what @MarsWarrior would like to have, to which I replied (to him) that a different solution would be more suitable for his needs, like the parser you mention!

Clarifying what I meant

I want to extend dsmr-reader in such a way that it supports listening on an MQTT topic (like the one mentioned before _home/power/dsmr/DEVICEID). The messages the MQTT Listener (subscriber) receives could be considered as "remote" telegrams. This looks a lot like what you did with the dsmr_api, except that the responsibilities are a little bit different.

Why I want to do this? Well, to explain that I have to explain a little bit about my intended setup (which goes beyond just the dsmr-reader part 😉).

I am trying to build a small sensor network at home, with sensors publishing their data to a central hub (MQTT broker). MQTT is a well known standard and I believe heavily used also in the Maker community for devices like the ESP8266 (sort of Arduino, but not the same and with builtin WiFi for only $2,-).

By publishing this data to this central hub, any listener can do something with the data. In programming also known as the publish-subscribe pattern. It encourages loose coupling between devices that generate data and processes that, well, process data. This loose coupling also creates space to add more/different processes to do something with the same data without a device having to submit the same data to multiple endpoints.

In my case my hub is a server running Ubuntu, NodeJS, Python, MongoDB, MariaDB, and the MQTT Broker implementation called Mosquitto but this could very well be the raspberry pi where the dsmr-reader software is running (because these brokers are primarily lightweight). My setup involves multiple ESP8266 devices, one with a specific P1 reader shield that reads in telegrams from my meter and publishes these messages to the MQTT broker. Others publish temperature, humidity, soil humidity levels, light levels, etc. Some of these devices can also be controlled by sending data to the MQTT broker on certain topics.

Supporting MQTT for reading in telegrams from an external device, in addition to supporting API calls, could help support a bigger network of "things" that all work with MQTT. I would love to help out with this if you were to add this as a feature. If not, I could always write a separate script that listens to the MQTT broker for DSMR messages and then writes them to the API on localhost. I just believe that others might want the same thing and integrating this feature would help them too.

I hope this helps in clarifying what I meant and for you to understand my vision.

Have a good weekend!

Thomas

[dennissiemensma]

Thanks for providing more input. I do now understand the vision you have and also the reason for it, to prevent sending the same data twice.

However, this downside is that adding such feature will add more complexity to the project. And move any of the loose couplings, you intend to have, into this project as well. That the exact reason why I do not see fit for other sources of data that are not sent via the API or builtin datalogger.

This project requires the raw telegram data for many features and integrating another source would limit this. Or even worse, break any MQTT implementation and disappoint the users of it. The data you provided in your initial comment would suffice, for some fields, but not all of it. Recently there was support added for phases, in the future I would also need the device serial numbers (#225). And at this very moment I have to build support for DSMR v5, which might introduce some new features as well.

So I hope you can understand my decision for not integrating such feature now. The project is stable for almost a year now, but far from finished. So I do think there will be a future for such integration, but that would require this project to be fully matured first. Fow now I will push this back to a later release.

I can still provide you input for the script discussed earlier. Please note that there is a raw telegram needed due to all the reasons above.

[thomasvnl]

I understand the downsides and respect your decision for not including it at the moment. I do want to add that the MQTT payload can contain all the fields from the DSMR message or even better, the DSMR string itself. MQTT doens't limit you in terms of contents. In my case my ESP8266 module has a parser on it (to be more precise, this one), therefore I've chosen to send a JSON string and let the parsing and CRC checking be done by the reading module.

I think I can manage with the documentation and issue #187 to create a script to parse data from MQTT into the API, but thanks anyway for being willing to help.

[jeroenubbink]

@dennissiemensma would you be open for a pull request that will actually implement this through an admin screen? Maybe with a message template that allows you to design a message the way you like it using variables. It would be unobtrusive and disabled by default.

I didn't go through the code myself yet but i think this would be a rather simple and elegant solution.

[dennissiemensma]

Thank you for providing your support. However both this ticket and #9 will be included in the v1.8 release, which I'm starting on shortly anyway. I'm currently wrapping up #230 for v1.7.

I do require some support later on here, since I'm no expert in MQTT anyway. As well as some testers.

There is a question I have for you: How do I choose the right format? Does a simply mapping of all model fields suffice? For example (from the API docs):

            {
                "id": 3593621,
                "timestamp": "2017-01-31T23:00:03Z",
                "electricity_delivered_1": "1596.234",
                "electricity_returned_1": "0.000",
                "electricity_delivered_2": "1484.761",
                "electricity_returned_2": "0.000",
                "electricity_currently_delivered": "0.075",
                "electricity_currently_returned": "0.000",
                "phase_currently_delivered_l1": "0.017",
                "phase_currently_delivered_l2": "0.058",
                "phase_currently_delivered_l3": "0.000",
                "extra_device_timestamp": "2017-01-31T22:00:00Z",
                "extra_device_delivered": "1835.904"
            }

Will posting that data to a MQTT endpoint work?

[jeroenubbink]

In theory you could provide that payload and yes clients will be able to receive that payload.

However i don't think you should choose the format, rather you should make it a template that people can design, maybe default to the raw dmsr string. It really all depends on how flexible the subscriber software is. Maybe it can only read the payload if it's in a certain format or maybe it will allow you to parse the payload on the subscriber side. If the software is really inflexible it might actually only read hardcoded topics and expect payloads to be in a certain format.

I would even go as far as to say, you might have multiple subscribers who are inflexible and bind to certain topics so you might even want to send out multiple messages in various formats.

I don't know if you ever used AMQP but MQTT is from what i understand somewhat of a simplification of the AMQP protocol, you publish a message with a topic and a payload to the MQTT broker and the clients who subscribed to a certain topic will receive the payload as a message.

I have whipped up an example in a few minutes using the paho-mqtt publish#single implementation, the mosquitto docker image (but you could just apt install mosquitto) and the default mosquitto_sub subscriber client (apt install mosquitto-clients) to read the message.

So given you have mosquitto running on localhost (either through docker or through a local server) you can do the following in an empty shell:

$ mosquitto_sub -t 'dsmr/raw'

This will subscribe the commandline mosquitto client to receive messages with the topic dmsr/raw

Then in another terminal i am in a virtualenv with the paho-mqtt module installed.

>>> import paho.mqtt.publish as publish >>> publish.single('dsmr/raw', payload='this can literally be any kind of string', client_id='dsmrreader')

Then when i go back to the terminal with mosquitto_sub running i see:

this can literally be any kind of string

I think in a basic scenario you would want a user through the dsmrreader admin to:

• Specify the MQTT broker hostname [mandatory]
• Specify the MQTT broker port [mandatory]
• Specify the topic to publish to [optional]
• Specify what the payload should look like [optional]
• Specify multiple topics with different payloads [optional]
• Specify how often a combination of topic/payload should be published onto mqtt. [optional]

If you need any further help please let me know. I'd be happy to provide it.

[dennissiemensma]

Thanks a lot for the explaination! I will try mosquitto and the paho client to see how things work. Defining a template seems the way to go, as there are now multiple examples prompted by different users in this ticket. I will continue later on in the next release.

[dennissiemensma]

I'm currently looking for a uniform, but simple way to allow users have different kind of MQTT messages containing the data they want. I do think however that it's quite difficult to have a 1-solution-for-all.

I'm currently thinking of:

• A setting to simply allow to send a raw incoming telegram string to a topic.

• A setting to send a pre-formatted (but editable) JSON string to a topic, i.e.:

  {
  "id": {{id}},
  "timestamp": "{{timestamp}}",
  "electricity_delivered_1": {{electricity_delivered_1}},
  "electricity_returned_1": {{electricity_returned_1}},
  "electricity_delivered_2": {{electricity_delivered_2}},
  "electricity_returned_2": {{electricity_returned_2}},
  "electricity_currently_delivered": {{electricity_currently_delivered}},
  "electricity_currently_returned": {{electricity_currently_returned}},
  "phase_currently_delivered_l1": {{phase_currently_delivered_l1}},
  "phase_currently_delivered_l2": {{phase_currently_delivered_l2}},
  "phase_currently_delivered_l3": {{phase_currently_delivered_l3}},
  "extra_device_timestamp": "{{extra_device_timestamp}}",
  "extra_device_delivered": {{extra_device_delivered}},
  }

• A setting to send all of the fields above to a separate (editable) topic. Configuration would be somewhat similar to the one above.

Any thought on this or what is needed to make it as configurable but simple as possible? Or any stuff that I don't need?

[jeroenubbink]

This looks like a very decent way of doing it. I've meanwhile looked into how openhab(2) and homeassistant read mqtt messages but they both seem to require 'custom' functionality to transform the message, be it through some configuration (openhab) or a transform function written in python (homeassistant).

I would just go with this for now as this is a great start that should cover probably 90% of all the use cases. If people start asking something that is not possible with this template i'm sure it will be trivial to add.

I'm not sure if you really need to be sending it in different formats under different topics, but it does add additional flexibility which i guess is a nice-to-have.

[dennissiemensma]

Okay thank you for your input. I will maintain the current draft then.

[dennissiemensma]

I've managed to finish the implementation for this. I've tested it all locally using the mosquitto_sub, but I rather have someone run it as well in a real MQTT setup.

Does anyone wants to give it a try?

[jeroenubbink]

That sounds awesome! I was planning on integrating it in my OpenHAB setup so i can give it a try but that could take a little while (2-3 weeks?). If nobody has responded in between i'll let you know the results.

[dennissiemensma]

Thank you, I will most likely just release it for now and add any additions required later via patches or a new version. I'm looking forward to hearing from you in a few weeks then.

[dennissiemensma]

I've added some documentation and screenshots as well, to give you guys an idea of how it works:

• NL: https://dsmr-reader.readthedocs.io/nl/development/mqtt.html
• EN: https://dsmr-reader.readthedocs.io/en/development/mqtt.html