Provide daemon configuration and sensors via MQTT subscription topic

[rlmalisz]

We'll be using these sensors to monitor outdoor plants, and because of range issues for BLE, will have multiple monitoring daemons. It'd be lovely to be able to inform the daemons at wakeup what sensors they "own" without having to modify the config.ini by hand every time a new sensor gets added or a potted plant gets moved from one coverage area to another.

[ThomDietrich]

I get your use case but didn't understand which alternative you are seeking.

Retrieving data from a database sounds like out of scope of the daemon. As a tradeoff, wouldn't it be acceptable to modify the config file via a helper script before starting the daemon?

[rlmalisz]

It would be possible to stop/update/restart, but if it’s multiple machines, some of which are running on solar and only waking up to take measurements, it might be awkward. I’m just arguing for a “soft” list of sensor devices. The entry in the config.ini file could be a list as it is now, or alternately, an external entity to invoke to populate the list before each measurement pass. It could return the list on stdout in the same format you use in the config file. So while I agree that the actual query is out of scope, support for allowing such a query would be a nice thing.

—Richard

On Apr 18, 2018, at 6:34 AM, Thomas Dietrich notifications@github.com wrote:

I get your use case but didn't understand which alternative you are seeking.

Retrieving data from a database sounds like out of scope of the daemon. As a tradeoff, wouldn't it be acceptable to modify the config file via a helper script before starting the daemon?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/35#issuecomment-382388108, or mute the thread https://github.com/notifications/unsubscribe-auth/ABN6VMVplOgqf9wro8b_olXOx9tJ2Zixks5tp0DygaJpZM4TZCaS.

[ThomDietrich]

I see.

Well okay, I feel your idea is a bit too specific. I like the idea to centralize the configuration point for multiple sensors. How about the following:

We are already using MQTT for data publication. We could support an MQTT configuration topic per client, so that every client would subscript to a list of sensors he is supposed to query.

A few more details would need to be discussed but I like the idea in general.

...sadly I am currently not able to work on this. Would you like to push the idea forward?

[rlmalisz]

So how well does this work if the endpoint reading the sensors is asleep much of the time, and is not actively subscribed at the time an update gets published? If this shows my ignorance of some of the nuances behind the pub/sub model MQTT supports, I apologize.

—Richard

On Apr 18, 2018, at 1:06 PM, Thomas Dietrich notifications@github.com wrote:

I see.

Well okay, I feel your idea is a bit too specific. I like the idea to centralize the configuration point for multiple sensors. How about the following:

We are already using MQTT for data publication. We could support an MQTT configuration topic per client, so that every client would subscript to a list of sensors he is supposed to query.

A few more details would need to be discussed but I like the idea in general.

...sadly I am currently not able to work on this. Would you like to push the idea forward?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/35#issuecomment-382512436, or mute the thread https://github.com/notifications/unsubscribe-auth/ABN6VIdvB7eJhPv28N0_fw66BJ3RbjXQks5tp5zPgaJpZM4TZCaS.

[ThomDietrich]

That shouldn't be an issue. Tell me when you'd expect a config update to take effect. Right now the config is only read on read during daemon startup. Would you expect it to be evaluated every time a new sensor readings update is scheduled? That would totally be possible, we just have to figure out the details.

[rlmalisz]

I think only the sensor list would need re-evaluated, which is why I was envisioning it optionally being provided by some hook, oracle, whatever. I was planning on getting readings every two hours, and should certainly be able to have the Pi Zero W wake up on a schedule that the central “provisioning daemon” would know about. Then that pushes a minute or so after scheduled wakeup, the minions wait for the info and then scan. I think, under that model, it’s fine if the mothership sends the sensor list on schedule irrespective of changes, at least, for the minions that spend a lot of time asleep. The minions could publish back their next planned wakeup.

I think it’s doable. I also think there’s still some merit in allowing the sensor list to be provided by an arbitrary helper as listed in the config file. Consider the (truly) degenerate case: the helper, when invoked, could simply do an lescan for a minute or two, and return the list with BTADDRS as both name and BTADDR. All the subscriber to the published data needs to know is the association of the BTADDR to the sensor and its plant or whatever. I move a pot from one area of coverage to another, and it just takes care of itself. The list could be a network-mounted file, and the helper just cats it.

I’m not suggesting making the daemon a lot more complicated. I’m just suggesting having the option of having the sensor list acquired by opening a pipe to some process or helper, instead of hard-coded. If the user chooses to manage that via MQTT and work out the timing, plumbing, etc, that’s fine. I’m in favor of delegating this to the user…all we need is to allow the hook, to invoke it before each set of readings if provided, and make sure the format the hook responds with is well-understood.

—Richard

On Apr 18, 2018, at 3:38 PM, Thomas Dietrich notifications@github.com wrote:

That shouldn't be an issue. Tell me when you'd expect an update to take effect. Right now the config is only read on read during daemon startup. Would you expect it to be evaluated every time a new sensor readings update is scheduled? That would totally be possible, we just have to figure out the details.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/35#issuecomment-382551152, or mute the thread https://github.com/notifications/unsubscribe-auth/ABN6VFl-0rHFDcYIe8FzpJaCzBgyIlP2ks5tp8BXgaJpZM4TZCaS.

[ThomDietrich]

Your idea of a directing mothership which will directly interact with clients will massively over-complicate things. I believe from the use cases I was able to derive from the basic idea of a distributed system we are looking for a daemon which can be easily installed on a, e.g. RPi, and then be mostly configured from remote.

I would definitely go with MQTT then. It is already used and the concept of retained messages is just great for this.

Conceptually it would look like this:

A daemon will go through the following cycle:

• Read the latest data published to its dedicated configuration topic (e.g. miflora/config/daemon_unique_id)
• Store the configuration in temporal memory
• Poll all sensors according to configuration
• Optionally: Scan for available sensors, publish to a dedicated topic.

With this behavior in place your central control entity can publish configs (including sensors, polling times, ...) for all clients. On top of that it will learn which client sees which sensor and can (help) rearrange authorities.

[rlmalisz]

Retained messages make this all pretty straightforward. I was not aware of them being part of the MQTT protocol. As I said before, I am new to MQTT.

—Richard

On Apr 18, 2018, at 4:18 PM, Thomas Dietrich notifications@github.com wrote:

Your idea of a directing mothership which will directly interact with clients will massively over-complicate things. I believe from the use cases I was able to derive from the basic idea of a distributed system we are looking for a daemon which can be easily installed on a, e.g. RPi, and then be mostly configured from remote.

I would definitely go with MQTT then. It is already used and the concept of retained messages is just great for this.

Conceptually it would look like this:

A daemon will go through the following cycle every x seconds (configured polling time):

Read the latest data published to its dedicated configuration topic (e.g. miflora/config/client-C3) (the actual update of data from this topic can happen when changed or at this point now) Store the configuration in temporal memory Poll all sensors according to configuration Optionally: Scan for available sensors, publish to a dedicated topic. With this behavior in place your central control entity can publish configs (including sensors, polling times, ...) for all clients. On top of that it will learn which client sees which sensor and can rearrange authorities.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/35#issuecomment-382558987, or mute the thread https://github.com/notifications/unsubscribe-auth/ABN6VBCMjsjdZKbYtgEGVAjQ3M1hpkgXks5tp8nZgaJpZM4TZCaS.

[rlmalisz]

I have caught with your latest/greatest and will start working on the “get sensor settings via MQTT” in a fork, so I don’t mess up life for others.

With the latest, I am seeing what I believe is new behavior: at startup, if one of the listed BTADDRs cannot be reached for the initial connection test, the Python script crashes. We moved some plants out of the greenhouse, and the first one in the sesnor settings is likely just a little too far from the BT dongle. The attempt to connect hangs for a good while, and then the script crashes, which is bizarre, since the attempt is in a try/except block. The crash info looks like:

Adding sensor to device list and testing connection ... Name: "Meyer_Lemon" Traceback (most recent call last): File "/opt/miflora-mqtt-daemon/miflora-mqtt-daemon.py", line 218, in flora_poller.fill_cache() File "/usr/local/lib/python3.5/dist-packages/miflora/miflora_poller.py", line 61, in fill_cache firmware_version = self.firmware_version() File "/usr/local/lib/python3.5/dist-packages/miflora/miflora_poller.py", line 104, in firmware_version res = connection.read_handle(_HANDLE_READ_VERSION_BATTERY) # pylint: disable=no-member File "/usr/local/lib/python3.5/dist-packages/btlewrap/gatttool.py", line 23, in _func_wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/btlewrap/gatttool.py", line 253, in read_handle raise BluetoothBackendException("Exit read_ble, no data ({})".format(current_thread())) btlewrap.base.BluetoothBackendException: Exit read_ble, no data (<_MainThread(MainThread, started -1220109952)>)

If I comment out this sensor, things proceed as expected. But I know before while I was trying to find the best position for the BT dongle that I was just getting no readings for the unreachables.

Just tried this again with a different (unreachable) sensor and system. Same crash. I’m sure you can replicate it by simply munging an octet in your BTADDR sesnor list.

—Richard

On Apr 18, 2018, at 5:29 PM, maliszewski richard rlmalisz@nguzu.com wrote:

Retained messages make this all pretty straightforward. I was not aware of them being part of the MQTT protocol. As I said before, I am new to MQTT.

—Richard

On Apr 18, 2018, at 4:18 PM, Thomas Dietrich <notifications@github.com mailto:notifications@github.com> wrote:

Your idea of a directing mothership which will directly interact with clients will massively over-complicate things. I believe from the use cases I was able to derive from the basic idea of a distributed system we are looking for a daemon which can be easily installed on a, e.g. RPi, and then be mostly configured from remote.

I would definitely go with MQTT then. It is already used and the concept of retained messages is just great for this.

Conceptually it would look like this:

A daemon will go through the following cycle every x seconds (configured polling time):

Read the latest data published to its dedicated configuration topic (e.g. miflora/config/client-C3) (the actual update of data from this topic can happen when changed or at this point now) Store the configuration in temporal memory Poll all sensors according to configuration Optionally: Scan for available sensors, publish to a dedicated topic. With this behavior in place your central control entity can publish configs (including sensors, polling times, ...) for all clients. On top of that it will learn which client sees which sensor and can rearrange authorities.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/35#issuecomment-382558987, or mute the thread https://github.com/notifications/unsubscribe-auth/ABN6VBCMjsjdZKbYtgEGVAjQ3M1hpkgXks5tp8nZgaJpZM4TZCaS.

[ThomDietrich]

39

[ThomDietrich]

@rlmalisz How is your fork working for you? https://github.com/rlmalisz/miflora-mqtt-daemon/commit/b689503926e4af20fa86d63f4256e59fda6370bf

[ThomDietrich]

@bangom this ticket is a bit older but still very relevant. Are you also interested in this? https://github.com/ThomDietrich/miflora-mqtt-daemon/issues/35#issuecomment-382558987

[rlmalisz]

I have no complaints about my fork. We have a half-dozen "readers" around the house and yard. Not too many sensors deployed this time of year, but in summer, a couple dozen. Readers collect their assigned sensor readings every three hours. Just before that interval, a cron (OSX, so actually launchctl) script queries a MySQL DB for assignments, and publishes them to MQTT. Readers' scripts run, get their assignments, and gather readings, and publish them. Another OSX scripts runs after those finish, and collects published readings and adds them to MySQL. It also posts warnings about any readings that are too low (or high) for sensors. Have a PHP script that uses Google Charts to display recent data queried from the DB. Image attached. But adding/deleting/moving a sensor is a simple update to a couple DB tables. This was what I wanted, so am quite happy.

[bangom]

@bangom this ticket is a bit older but still very relevant. Are you also interested in this? #35 (comment)

Most definitely. Next miflora-mqtt-daemon development step should be implementing distributed miflora-mqtt-daemons configuration from single place (master node?). This is IMHO primary usecase for miflora-mqtt-daemon (multiple btle2wifi clusters of RPI0 connected to central hub like Home Assistant). Let me think this over and get back with my ideas :) But from the initial analysis of this issue I like the idea of using MQTT (sensor data and daemons configuration updates).

[ThomDietrich]

I already have an idea. Let me post a proposal this evening