Add basic mqtt support

[sfudeus]

This adds basic MQTT support.

When I started playing around with HomeAssistant I tried to find an easy way to integrate the data from the RCT inverter. Having this data already in prometheus, one option was to read it from prometheus - but this is not an ideal integration with HomeAssistant. Another option was an additional bridge between RCT and MQTT, using the rctclient directly. But each individual client against the inverter produces more interfering traffic, so the natural thing seemed to be to just let rctmon emit data to an additional backend. For the sake of simplicity, the MQTT topic names are just derived from the promtheus metric names. This does not always end in ideal or intuitive names, but it works well enough.

[rayb67]

Hi Stephan, I found this thread and I don't know if this is the right place for question/discussion. I am looking for a docker based rctmon implementation, where a mqtt solution is in use. Because I have openhab + mqtt up a running, I use the rctclient solution installed direct in the linux. My current setup: Based on the battery.py (I don't know where I download it), use an extra shell script, scheduled by cron, which call my rctmon.py, to read the PV parameter and put it into mqtt. But I am on the way to use docker for all. I use the shell script, because of no experience with python. But, the script work excellent over more than one year. Very important is, the python script must run exclusive and only one instance. Therefore I check if a session is running and kill that.

rctmon.bash:

#!/bin/bash
kill -9 `ps -ef | grep rctmon.py | cut -c10-15`
/usr/bin/python /usr/local/bin/rctmon.py

My future ? Based on a docker container you provide, I will read the PV and put it into the mqtt.

a) you add these two files into your container and crate a cron entry for the bash very 5 minutes. I can add the rctmon.py to the docker container call as parameter to control the information I need. b) I go on with your way. But I need a little more information / documentation. How can I enable the mqtt support? Can I control the mqtt IP ?

Best regards Ralf

[sfudeus]

Feel free to try option (b), the documentation about how to enable mqtt with this PR can be found in the config.example.yml and docs/use_overview.rst in this pull request. You can override the mqtt_host and mqtt_port. Setting the mqtt_host in fact is mandatory. You can find a docker container already containing this enhancement at docker.io/sfudeus/rctmon:preview-preview. With this image, just mount your config.yml as /config.yaml. Important parts to set are

device:
  host: $your_inverter_hostname
  port: 8899
mqtt:
  enable: true
  mqtt_host: $your_mqtt_hostname

If you see issues or don't get it up and running, feel free to ask.

[rayb67]

Hi Stephan, I try it and it is working. Great job !

I have two question/request. a) Can we control the frequency calling the rct inverter via parameter in the config.yml file? b) Can we have "a" rctmon.log file linkt into the local filesystem like config.yml?

Thx Ralf

[rayb67]

Hi Stephan, I check some values from my current solution and search them yours.

I miss some. Some of them I detect as WARNING in the dockers console log .

e.g.

2023-08-20 09:11:18,655 - rctmon.device_manager - WARNING - Index 0x2f3c1d7d not in frames list
2023-08-20 09:03:28,732 - rctmon.device_manager - WARNING - _cb_energy: unhandled oid 0x10970E9D

10970E9D > This month's energy [Wh] 2f3c1d7d > Today's energy consumption household in Wh

How can we fix the WANING and handle the "old" list as well as the "...not in frames list..."

Best regards Ralf

[sfudeus]

@rayb67 a) querying frequency: Should be easy enough, currently this is a static value AFAIC. b) rctmon.log: The container currently emits on stdout. IIRC logging is configurable already, so if you mount a directory into the container and configure the logging stack to write the logs into that directory, this ought to work, but I need to check myself to be sure. c) The missing values should be easy to add. The total energy counters should be emitted already (in contrast to daily or monthly data). Most folks likely use those, so nobody missed them yet.

fyi: If I implement a and c, they'll go into distinctr pull requests, since different features. I'll mention them here and they likely go into my preview image then as well.

[sfudeus]

@rayb67 regarding (a): I rechecked, this is more difficult, because this is already (statically) dependend on the measured item and its expected volatility. Some values are checked every 10s, some once per minute, some only every 5m. What do you want to achieve with you request? Likely this makes sense to register as a dedicated issue in this project to discuss specifically.

[poggenpower]

@rayb67 have you considered to add authentication and TLS support to the mqtt connection. Here what I would suggest to add: https://github.com/svalouch/rctmon/commit/b20f7acdf8cc754ee4db2cd99f1af9d48cc58054

[sfudeus]

@poggenpower great, I added a PR to my original branch to add your contribution right into this. (https://github.com/sfudeus/rctmon/pull/1)

[sfudeus]

fyi: I just rebuilt the docker.io/sfudeus/rctmon:preview-preview image with the latest changes

[rayb67]

@rayb67 have you considered to add authentication and TLS support to the mqtt connection. Here what I would suggest to add: b20f7ac

TLS is not helpful for me. I think it is overhead in a privat environment. My focus is on the function, not at security. A trusted communication in this case too much. There are no data, which I think we need to protect.

[rayb67]

@rayb67 regarding (a): I rechecked, this is more difficult, because this is already (statically) dependend on the measured item and its expected volatility. Some values are checked every 10s, some once per minute, some only every 5m. What do you want to achieve with you request? Likely this makes sense to register as a dedicated issue in this project to discuss specifically.

Reduce the traffic and the load. Values per 10 seconds. Which use case need these amount of data? OK, I have less experience with mqtt protocol. With openhab , I store these data in longer terms, but it is not necessary store these amount data .

I will try to set configure the log file and to on with my test. Thx for the support ;-)

[svalouch]

@rayb67 Data reduction: I've commented on that in general in #19. Log file: Python is very flexible in that regard, the default config is in logger.py. Simply include a top-level element named logging in the config file to replace the default from the file. You can find information on that over in the python-docs: Configuration dictionary schema. No code changes should be required for most use-cases.

@sfudeus Cool feature!

Few ideas on this:

1. Wouldn't it make sense to expose the value store from within DeviceManager to the outputs? Or is simply working off of the Prometheus metrics actually easier here?
2. MQTT allows the most recent value to be stored, so perhaps it makes sense to not push the whole nine yards each time, but only what's changing and set the persistence bit plus a "last will"? New clients would get the entire set on connect from the server without rctmon having to do anything, and then would just get the changes as they fly in, like the InfluxDB output. That should be less volume, especially in bulk, which I imagine could be a problem for things like an ESP8266 (but that's just gut feeling, I haven't tested that and it may not be a problem at all).

But I'd be fine merging this as-is as well, let me know!

[sfudeus]

@svalouch Hmm interesting idea - I originally chose the prometheus data because of the existing metadata. But theoretically I can as well just use the property names via reflection. You thought about something like this on all the model classes?

    def __setattr__(self, name, value):
        old = getattr(self, name)
        if old != value:
            log.debug("change for {} detected, old {}, new {}".
                      format(name, old, value))
            self.__dict__[name] = value
            # TODO: submit via mqtt if enabled

or just register a changelog which is then flushed and reset periodically to be able to batch submit to mqtt?

[rayb67]

@svalouch belongs to data reduction. It took a little time for me to understand how to handle the mqtt and openHAB . But everything works fine. The concept you explain is fine. I can control the amount of data in openHAB independent from the mqtt setting.

[poggenpower]

@rayb67 have you considered to add authentication and TLS support to the mqtt connection. Here what I would suggest to add: b20f7ac

TLS is not helpful for me. I think it is overhead in a privat environment. My focus is on the function, not at security. A trusted communication in this case too much. There are no data, which I think we need to protect.

@rayb67 I would really appreciate if you look into the work I have made on top of your code. It is all there. And merge it. Should not be a big thing. I understand, that you don't have the TLS use case, but other have.

[sfudeus]

Hey @poggenpower , your auth suggestions have already been incorporated into my branch/pull request, see https://github.com/svalouch/rctmon/pull/32#issuecomment-1693082970. Also I am not sure what code of @rayb67 you mean, I suppose you mean mine.

[poggenpower]

Hey @poggenpower , your auth suggestions have already been incorporated into my branch/pull request, see #32 (comment). Also I am not sure what code of @rayb67 you mean, I suppose you mean mine.

Sorry I got a little confused here. I thought rayb67 is owning the PR. Ignore my comment thanks for your work!

[sfudeus]

closing this for the sake of #33