Add MQTT support

[andig]

Instead of adding yet more logging targets (e.g. https://github.com/volkszaehler/vzlogger/pull/300) I'd suggest to add something generic like MQTT. Other project like e.g. ebusd are using the same approach (https://github.com/john30/ebusd/commit/e82a3a051dead52d0df8796e3e70586c65c43735).

The MQTT target should be globally configurable (Uri, potentially user and password) as well as locally able to override. Topic name should be configurable per channel (or per meter with channel automatically appended).

A topic might by default look something like vzlogger/meter0/1.8.0.

It would probably make sense if publishing topics uses the same logic as push updates. Aggregated updates might be output to vzlogger/meter0/1.8.0/aggregate in order to make the aggregation capabilities of vzlogger reusable.

[Sineos]

+1 for this idea. I'm currently doing this via Node-Red.

Depending on the usage and target of the MQTT telegrams, it would be great to have some sort of "templating" option in the API to control how the message is send, e.g.

        {
          "api": "mqtt",
          "uuid": "edc95ef0-dfd0-11e6-a86b-5d27a2655935",
          "identifier": "Power",
          "host": "http://192.168.178.8:1883",
          "topic": "vzlogger/blabla",
          "message": "value = $value, time = $timestamp, uuid = $uuid"
        }

Where in the message element everything is treated as string except some defined variables like $value etc, which are replaced by their actual content. This would allow tuning the message to exactly the format that is expected by the recipient of the telegram.

[Stefan-Code]

In retrospect, this would've been a better feature to implement than direct InfluxDB support...

[andig]

This would allow tuning the message to exactly the format that is expected by the recipient of the telegram.

Wouldn't do that. The principle should be separation of duties:

• vzlogger sends native data to mqtt
• mqtt distributes
• client subscribes and parses

Node-red can always be inserted in the middle of this chain.

[Sineos]

I'm not sure, if I get your point @andig. What would a telegram contain in your view? Just the value or value + timestamp or value + timestamp + uuid? If more than one value is send via a telegram, the "sender" already decides for some kind of template. Are the values json / xml or separated by comma, semicolon, etc.

The DIY home automation nerds I know (including of course myself) live in a very inhomogeneous world. An Arduiono here, a Homematic there, some Phillips Hue, 2 different vzlogger installs, some modbus stuff ... My biggest challenge is interfaces and how to convince all these components to talk to each other. I have one RPi with Node-red dedicated to only do parsing and "gate-waying". I'm thankful for each component in which I can influence the data / message format in a way that allows for direct processing without further needs of parsing and modifying.

[mbehr1]

I like the idea of finding a good, portable, flexible way to connect e.g. to OpenHab2. But if we implement such a way we should try to ease configuration as well. E.g. we should be able to announce whatever data we can offer but publish only the data that somebody needs

If we talk to a mqtt broker imho we can't achieve that. We could publish everything but this would be too much data. If we restrict it to whatever is configured we still need our old-fashioned config files,...

If vzlogger would be a mqtt broker this could be met (e.g. publishing all available data sources but distributing only if a client has subscribed for it). By this way openhab could autodiscover all data sources from a vz installation. (On the other hand I think mqtt is not designed that each service is his own broker. Do I miss something here?)

[frankrichter]

Afaik MQTT ist not really designed to have multiple brokers. If there were multiple vzlogger instances in a setup (like mine), the clients would have to subscribe to different brokers to get all the data.

And I think a zero configuration approach is not really suitable for vzlogger. As mbehr1 said, this would result in transmitting reasonable amounts of potentially unnecessary data, and we would give up quite a bit of the achieved flexibility.

But what about interactive configuration, e.g. a lightweight web interface or something like this? For example we could list all available OBIS identifiers or all available 1-Wire sensors with their respective values to make channel configuration much easier.

[justinotherguy]

Afaik MQTT ist not really designed to have multiple brokers that's the my understanding, too.

If vzlogger would be a mqtt broker this could be met (e.g. publishing all available data sources but distributing only if a client has subscribed for it). By this way openhab could autodiscover all data sources from a vz installation the way to go might be this (basically what @andig wrote):

• we implement an output (currently all called "apis") for mqtt
• we publish all defined channels to the broker
• a subscriber can then discover all the channels and subscribe to the channel he's interested in -> this should be compatible to the concept of mqtt, should allow openhab (and possibly others) to autodiscover the channels It would require some new logic for the channel handling, though.

[Sineos]

The just released MQTT v5 has some interesting new features. For vzlogger the most useful would be properties in addition to payload. Ref: https://paolopatierno.wordpress.com/2017/08/21/mqtt-v5-what-is-on-the-way/

[andig]

What would a telegram contain in your view? Just the value or value + timestamp or value + timestamp + uuid?

In terms of payload just the value if we can assume that MQTT is online. If vzlogger's buffering applies here payload plus timestamp. Shouldn't be necessary with MQTT though.

Publishing goes to:

volkszaehler/<uuid>

maybe to

volkszaehler/<uuid>/current

or similar to get "power" values like the frontend API provides. Rather not though.

The volkszaehler prefix must be configurable to run multiple loggers against a single broker.

If more than one value is send via a telegram, the "sender" already decides for some kind of template. My biggest challenge is interfaces and how to convince all these components to talk to each other.

I don't see this task of vzlogger. Neither is e.g. influxdb integration.

Are the values json / xml or separated by comma, semicolon, etc.

Plain value. Anyway this is what e.g. ebusd does (https://github.com/john30/ebusd/wiki/3.3.-MQTT-client).

announce whatever data we can offer but publish only the data that somebody needs

The MQTT principle is more to publish everything and let clients decide what they subscribe to. Why restrict on the publishing side?

If we talk to a mqtt broker imho we can't achieve that. We could publish everything but this would be too much data. If we restrict it to whatever is configured we still need our old-fashioned config files,...

Why? If there are no subscribers the broker will ignore the data.

Afaik MQTT ist not really designed to have multiple brokers. If there were multiple vzlogger instances in a setup (like mine), the clients would have to subscribe to different brokers to get all the data.

That my understanding as well. Exactly as Justin summarized above.

[frankrichter]

we publish all defined channels to the broker

This is very similar to the way the existing push output works. Perhaps a good starting point for an easy implementation?

[Sineos]

Publishing goes to:

volkszaehler/uuid

maybe to

volkszaehler/uuid/current

Please consider to have the topic defined by the user. Subscribing to uuid is complicated because you always have to look them up in the config file to match it to the actual meter.

[andig]

Please consider to have the topic defined by the user. Subscribing to uuid is complicated because you always have to look them up in the config file to match it to the actual meter.

Good point. vzlogger doesn't actually know the uuids by itself so it's not quite natural using them. It does know meters and probably the kind of measurements they produce- either implicit (S0) or explicit (e.g. OBIS codes for SML).

So vzlogger should rather publish to

volkszaehler/meter1/1.8.1

and

volkszaehler/meter2/impulse
volkszaehler/meter2/current

or similar?

[bgewehr]

I see another approach. MQTT built in into vzlogger would be very useful to subscribe to readings coming from the broker and as well send meter values or impulses “nakedly” to the broker (a user defined message structure surely helps to adapt to target expectations) - but the magic of volkszaehler is the middleware and the ability to calculate consumption and momentanverbrauch from different sources.

I would like to see a middleware component, sending calculated data in selected periods to a broker so the target service does not again have to collect and calculate consumption or momentanverbrauch from e. g. impulses.

This way we get as much precision as the meter can deliver and as much elegance to attach 3rd party services with low network impact and effort in the target service. But I will have to post that to the volkszaehler repo, right?

[bgewehr]

https://github.com/volkszaehler/vzlogger/issues/238

[andig]

Versteh ich nicht. Vzlogger und MW sollen beiden zum Broker senden? Warum?

[bgewehr]

Die MW kann Momentanverbrauch zum Broker senden, der vzlogger manchmal nur Impulse. Die MW kann consumption24h oder sowas zum Broker senden, vzlogger nicht. Praktisch, wenn ich die Middlewareberechnungen nicht an anderer Stelle erneut nachbilden möchte.

[andig]

Die MW kann Momentanverbrauch zum Broker senden, der vzlogger manchmal nur Impulse.

Der Logger kann auch "power", das erscheint mir auch sinnvoll- einmal Rohwerte, einmal Momentanwerte.

Die MW kann consumption24h oder sowas zum Broker senden, vzlogger nicht. Praktisch, wenn ich die Middlewareberechnungen nicht an anderer Stelle erneut nachbilden möchte.

Ja, hat im Logger aber nichts zu suchen und kann einfach in der MW abgefragt werden.

Es macht keinen Sinn eine Monsteranwendung (=Logger) noch monströser zu machen. Stattdessen auf den Anwendungsfall fokussierte Funktionen.

[sir106]

Ich fände das Feature auch genial! von mir ein +1 für mqtt messages aus dem VZLOGGER....

[tunip]

Auch ich würde eine direkte MQTT Integration in den vzlogger begrüßen 👍

[mbehr1]

Also: ich würde jetzt mal in vzlogger mit der Implementierung anfangen. Gibt es Wünsche bzgl. der zu nutzenden Library? libmosquitto oder paho stehen auf den ersten Blick zur Auswahl. Erfahrungen? Wünsche? Sonst wähle ich einfach eine aus.

[andig]

Moin Matthias,

schau Dir mal ebusd an- C Code mit integriertem MQTT.

Ansonsten:

Es macht keinen Sinn eine Monsteranwendung (=Logger) noch monströser zu machen. Stattdessen auf den Anwendungsfall fokussierte Funktionen.

Ich fände es langfristig wichtiger den vzlogger in einfachere Funktionen zu zerlegen und die Gesamtarchitektur flexibler zu machen. Aber das ist eine ganz andere Nummer...

Viele Grüße, Andreas

[mbehr1]

@andig ebusd nutzt libmosquitto. Wäre ok für mich. Bzgl. "flexibler/einfacher". stimme dir voll zu. Sollten grundsätzlich nur noch generisch/abstrahierende Funktionen integrieren. Bei mqtt sehe ich das prinzipiell. Wobei für mich eine Umsetzung in der MW oder ein "push-2-mqtt" Modul auch ok wäre. Daher mache ich mir auch eher Sorgen um die Konfiguration und würde nach Möglichkeiten suchen, mqtt mit so wenig Konfig wie möglich zu integrieren (z.B. per default alle Daten pushen) umzusetzen.

Bin für andere Ideen offen.

[tunip]

libmosquitto klingt gut.

[mbehr1]

see pr #357 for further discussion