Optionally generate separete MQTT messages for additional sensors

[wolfgangr]

Problem My weather station reports not only it's weather sensors, but also other sensors in proximity compatible to the "ecowitt ecosystem", e.g. in my case some WH31 of a HP3000 clone coined "ELV RoomLogg": "temp1": 20.777777777777782, "humidity1": 51.0, "batt1": "OFF",

Consider the well known ecowitt sensor mapping:
There is a growing number of possbile extension and combiniations of sensors.

Not all their data flow into the the same destination.
Adding sensors to existing configurations may result in complex configuration changes in several points.
Frequent long messages may push lightweight receivers beyond their limit.
So it increases flexibility if we generate independent MQTT messages with independent configured topics and frequency.
In terms of database design, we might talk about those extre sensors comprising different entities.

Preferred solution in ecowitt2mqtt Can we expand the configuration and processing in a way that

• additional sensors may be included or omitted in the main message of the station
• extra sensors may generate seperate MQTT meassages with configurable topic, frequency
• new sensor instances and new sensor types can added "on the fly" by configuration changes

Alternative approaches:

• dedicated Gateways
• generating additional events in e.g. my database logging script
• running a dedicated process that subscribes to all sensor data an generates messages for extra sensors
• using skd of home automation system for this purpose

[wolfgangr]

The German wiki page for MQTT distincts between

• the broker, where all relevant data of a system is kept
• silly (aka lightweight) clients, which just supply sensor data and recive simple commands
• intelligent/powerful control instances performing complicated data integration

Gleaning over the source, I'm quite sure that ecowitt2mqtt had the intelligent power to implement the configurable separation of data. Unfortunately, I'm afraid that I am to silly to unveil this power.

So, unless there is neither any sign of need nor support from the ecowitt2mqtt comunity, I'd prefer the "works for me" way keep ecowitt2mqtt as simple interface layer to some simple weather station and implement the separation of Auxilariy Sensors in my PERL based database logger.

However, If there is need and / or support, I might head towards a fork with pull request in the end.

[wolfgangr]

I've implemented it in PERL now, along with the database logger.
If somebody wants to implement it in ecowitt2mysql (or whatever), it may serve as pseudocode at least ;-).

The code processing the aux sensors starts here: https://github.com/wolfgangr/ew_mq_log/blob/effb0b656b93252f7d090f8be963cf738bddfe31/log_aux_ecowitt.pl#L95

Now my data flow is as follows:

Station and sensors 
   V--- RF transmission (16 ... 64 s)
Gateway / weather station
   V--- custom gateway ecowitt format (set to 10s)
ecowitt2mqtt (incl. dateutc hack)
   V--- mqtt (processed incl dateutc)
sub parse_ecowitt
   +--> sub log_ecowitt --->  SQL raw table of station outdoor every 16 s
   V--- sub parse_aux (extracts available temp/hum aux per sensor data)
sub do_aux (filters only readings per sensor where values have changed)
   +---> sub log_aux ---> SQL aux_th table with changed readings of every sensor
   V
sub pub_aux (filter configurable change bandwith)
   V--- mqtt retain - payload per sensor
lightweight consumers (TBD)

Bandwith funnel:

• Raw logging is performed 4 times per minute
  system load is barely noticeable at a 4core old pentium running debian
  1,5 GB storage in 10 years
• 5 aux sensor produce ~ 1 ... 2 readings per minute if we omit readings unchanged
  (instead of 4 * 5 = 20 per minute for the unfiltered case)
• bandwith filtering reduces to 0.3 ... 1 reading / MQTT pub per minute

Data generated with 1 aux sensor outdoor, 2 indoor and 2 greenhouses at a sunny morning.
Pub message reason config:

• temp change >= 1 °C
• humidity change >= 5 %
• last message older than 10 min
• sensor conected

Publishing data as "retained" keeps state of last reading at the broker. Thus a connecting/ polling subscriber gets data without waiting for a change event.

Example readings:

...$ mosquitto_sub -t wetter/pleussen/aux/# -h my.server.local -v
wetter/pleussen/aux/1/0 { "dateutc" : "2024-08-26 08:37:10" , "temp" : "24" , "humidity" : "53" , "batt" : "" , "reason" : "online" }
wetter/pleussen/aux/1/1 { "dateutc" : "2024-08-26 08:40:38" , "temp" : "29.1111111111111" , "humidity" : "68" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/3 { "dateutc" : "2024-08-26 08:41:26" , "temp" : "33.8888888888889" , "humidity" : "58" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/6 { "dateutc" : "2024-08-26 08:37:10" , "temp" : "18" , "humidity" : "73" , "batt" : "OFF" , "reason" : "online" }
wetter/pleussen/aux/1/4 { "dateutc" : "2024-08-26 08:37:10" , "temp" : "24" , "humidity" : "53" , "batt" : "OFF" , "reason" : "online" }
wetter/pleussen/aux/1/2 { "dateutc" : "2024-08-26 08:40:54" , "temp" : "18.3888888888889" , "humidity" : "72" , "batt" : "OFF" , "reason" : "online" }
wetter/pleussen/aux/1/2 { "dateutc" : "2024-08-26 08:43:02" , "temp" : "18.7777777777778" , "humidity" : "69" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/1 { "dateutc" : "2024-08-26 08:46:46" , "temp" : "29.8888888888889" , "humidity" : "64" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/0 { "dateutc" : "2024-08-26 08:48:22" , "temp" : "24" , "humidity" : "54" , "batt" : "" , "reason" : "max_time" }
wetter/pleussen/aux/1/1 { "dateutc" : "2024-08-26 08:48:54" , "temp" : "30.1111111111111" , "humidity" : "67" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/2 { "dateutc" : "2024-08-26 08:52:22" , "temp" : "19.2222222222222" , "humidity" : "65" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/3 { "dateutc" : "2024-08-26 08:52:54" , "temp" : "34" , "humidity" : "56" , "batt" : "OFF" , "reason" : "max_time" }
wetter/pleussen/aux/1/1 { "dateutc" : "2024-08-26 08:55:49" , "temp" : "31.1111111111111" , "humidity" : "67" , "batt" : "OFF" , "reason" : "temp_change" }
wetter/pleussen/aux/1/2 { "dateutc" : "2024-08-26 08:59:34" , "temp" : "19.5" , "humidity" : "62" , "batt" : "OFF" , "reason" : "hum_change" }
wetter/pleussen/aux/1/0 { "dateutc" : "2024-08-26 09:00:21" , "temp" : "24" , "humidity" : "54" , "batt" : "" , "reason" : "max_time" }
wetter/pleussen/aux/1/3 { "dateutc" : "2024-08-26 09:03:17" , "temp" : "34" , "humidity" : "53" , "batt" : "OFF" , "reason" : "max_time" }
wetter/pleussen/aux/1/1 { "dateutc" : "2024-08-26 09:05:57" , "temp" : "32.1111111111111" , "humidity" : "58" , "batt" : "OFF" , "reason" : "max_time" }
wetter/pleussen/aux/1/1 { "dateutc" : "2024-08-26 09:08:05" , "temp" : "32.1111111111111" , "humidity" : "55" , "batt" : "OFF" , "reason" : "hum_change" }

[bachya]

Would you say this overlaps with https://github.com/bachya/ecowitt2mqtt/issues/1073? If so, I'd like to consolidate all conversation over there.

[wolfgangr]

In my (user) view, it loosely relates, but not really overlaps.
I'd consider putting a crossref in #1073 to #1080, too, but not really merge it.

but... well..., do you plan to implement it? both?
Adding calculated values to the station reading even more blows up the message.
So, keeping aux sensors in seperate messages might ease this.
And when digging in the code of aux sensors, may be for the programmer it's more efficient to handle both proposals in one run.

In the end, it's your project :-), you decide.

[bachya]

but... well..., do you plan to implement it? both?

I haven't decided yet. It isn't clear to me how this would be accommodated in a simple fashion that correlates to ecowitt2mqtt's design and purpose.

[wolfgangr]

I perfectly understand.
That's what I've read from the MQTT "philosophy" in the German wiki:
Keep ecowitt2mqtt as a lean and tidy sensor interface.

Why not close the issue and wait whether any other users request the feature as well?

[bachya]

Okay, sounds good! Appreciate the content.