search

ThomDietrich / miflora-mqtt-daemon

Linux service to collect and transfer Xiaomi Mi Flora plant sensor data via MQTT to your smart home system, with cluster support 🌱🌼🥀🏡🌳
MIT License
606 stars 141 forks source link

Historic data for timeseries databases #89

Open andibraeu opened 4 years ago

andibraeu commented 4 years ago

The miflora poller provides the ability to read history data from our plant sensors. I wonder if it is possible to make use of this feature? I feed my openhab2, and the data is persisted in an influxDB. It looks that it is not possible to store a different timestamp for a value. So it could be hard to store bulk history data. Maybe some of you know better?

When I used the default poll interval (5 min) the batteries got too low after about 8 to 10 weeks. Now I extended the poll interval to 3 hours, but I still like to see the more detailed data.

ThomDietrich commented 4 years ago

The primary use case of this daemon is to provide data to smart home systems. They wouldn't benefit from historic data but I get your point about a direct path into InfluxDB or similar. I am not sure if this is out of scope of the future of this software.

Feel free to continue discussion or propose a solution / PR

andibraeu commented 4 years ago

I see a use case also for historic data, as I use it in my openhab frontend.

I built a dashboard to monitor my plants. At the moment I query my sensors every 3 hours, and the resolution of data points is quite low. image

It's not an option to have sensor queries more often, as I don't want to change batteries more often.

On the other hand I'm note sure if it would even be possible to add more data than the data we see during a query.

ThomDietrich commented 4 years ago

Your arguments are correct.

What I wanted to express: This kind of use case is not directly linked to the function of a smart home system. Of course you can pipe historic data to InfluxDB or any other kind of database or data store and then generate diagrams or insights from it - which is then presented in openHAB - but it's only tangentially relevant to the purpose of a smart home system being "automation of stuff"

ThomDietrich commented 4 years ago

To add one more though here: Retrieving historic data from the sensors is not a bad idea but MQTT would be the wrong interface to transfer them to. I saw a fork of this project by someone who then exchanged MQTT by a direct InfluxDB link. That would be the way to go.

NorbertHD commented 4 years ago

I saw a fork of this project by someone who then exchanged MQTT by a direct InfluxDB link.

@ThomDietrich Your project shows 83 forks. Do you have a link to the fork with the InfluxDB change?

ThomDietrich commented 4 years ago

Hey @NorbertHD, maybe @michaelbeljaars can help: https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/120

michaelbeljaars commented 4 years ago

@NorbertHD I have just uploaded my fork of @ThomDietrich's script (https://github.com/michaelbeljaars/miflora-mqtt-daemon) It has a section for InfluxDB in the config.ini. With current implementation it is not possible to use both MQTT and InfluxDB (not sure if that is even something you would want).

Additionally, when using Influxdb, the script also calculates and an average temperature for all the valid measurement and sends it to the measurement 'Average' in the database. I use this functionality to get an idea of the mean temperature in my apartment.

I still need to adjust the readme with all details, but be aware that on a RPi there have been problems with the bluetooth adapter. I have recently solved this by installing the latest version of Bluez (5.54) and installing the latest firmware (4.19.118-v7+) (See issue 120)

andibraeu commented 4 years ago

nice! :) thanks for sharing!

NorbertHD commented 4 years ago

Thank you @michaelbeljaars for sharing! I looked briefly through your code and it seems not to read the historic data. It reads one current dataset and sends it directly to the influxdb database in a loop. Right?

I have found a project on github that reads the historic data and sends it to an infuxdb database: https://github.com/sergem155/miflora-influxdb

andibraeu commented 4 years ago

yes, it doesn't seem to query historic data, but that's a good start to add historic data. The underlying library already supports historic data.

michaelbeljaars commented 3 years ago

Thank you @michaelbeljaars for sharing! I looked briefly through your code and it seems not to read the historic data. It reads one current dataset and sends it directly to the influxdb database in a loop. Right?

I have found a project on github that reads the historic data and sends it to an infuxdb database: https://github.com/sergem155/miflora-influxdb

Did you succeed in getting this to work reliably? I am confronted with many of these errors: connection and/or query failed - unable to connect?

Also, does anyone know with which frequency the sensors store measurements (i.e. how many per hour)? The implementation of the project above seems to indicate that this is only done once per hour.