tbnobody / OpenDTU

Software for ESP32 to talk to Hoymiles/TSUN/Solenso Inverters
GNU General Public License v2.0
1.77k stars 494 forks source link

[Request] Add livedata MQTT topic or send livedata to websocket server #1845

Open phol opened 5 months ago

phol commented 5 months ago

Is your feature request related to a problem? Please describe.

I'm logging the data produced by OpenDTU into a database using a small tool I wrote and open sourced yesterday, opendtu-logger. To do this, I'm using the websocket, as it contains a data structure more suitable for me than the MQTT topics. Whereas currently, the MQTT implementation is key-value based, I prefer to store historical data into normalised SQL tables as this can be much more efficient.

I initially built a proof-of-concept OpenDTU Logger using MQTT, storing data as "timestamp", "topic", "value". This works but is very inefficient.
With my new tool, I can store data as "timestamp", "value1", "value2", "valueN...". This is much more efficient in terms of storage and also makes querying data much more convenient.

OpenDTU Logger currently subscribes to the OpenDTU websocket as a client. However, this approach has a few drawbacks:

Describe the solution you'd like

I'd like to request the addition of a new MQTT topic, dtu/livedata, which I could use instead of the websocket.

This would fix all 3 listed drawbacks.

To this topic, the OpenDTU would only have to push the exact same contents as it currently pushes to the websocket. I don't mind if in a future release, the structure would change, as my tool is specifically developed to work with OpenDTU and can be updated again if required.

Alternatively, I'd like OpenDTU to be able to subscribe to a websocket server as a client, and push the livedata contents to that. This way, OpenDTU Logger doesn't need to connect to the OpenDTU which might be behind a firewall.

Describe alternatives you've considered

An alternative I considered was to request the addition of an "offline" notification in the websocket, but a new MQTT topic seems like a better solution to me as it would be more robust and also cover cases such as sudden power loss.

An added benefit of the MQTT topic is that future OpenDTU GUIs could be run standalone, on a different network, and could show data based on a MQTT subscription as Vue supports MQTT over websockets. With a livedata MQTT topic, the ESP32 is no longer the limiting factor in terms of the maximum number of clients supported, as much of the load is shifted to the MQTT server.

Another alternative I considered was to request including the possibility to connect OpenDTU to my logger using a WireGuard connection, but this would basically be a giant workaround. Moreover, a pull request to include WireGuard was denied, so I think the approach above would be better.

Additional context

OpenDTU Logger is available in a git repository: OpenDTU logger To explain a bit of background and my motivation to write this tool, I wrote a blog post.

HugoLS commented 1 month ago

Hello @phol Thank you for highlighting this because i got the exact same problem/need.

it could be great if opendtu was able to export livedata through mqtt.