Immediate update of throttled_average if large delta detected

[ColColonCleaner]

Describe the problem you have/What new integration you would like

Getting smoothed and throttled readings on a sensor while still allowing instant updates for large changes.

Please describe your use case for this integration and alternatives you've tried:

I don't currently see a way to smooth and throttle values from a sensor, while still allowing instant updates if a large delta happens.

I tried doing this with an 'or' filter containing a delta and throttle_average:

or:
  - delta: 20.0
  - throttle_average: 5s

But the result is the below graph, where the sensor seems to flip between the delta and the average whenever a large change happens, or at least this is what seems to be happening.

If there isn't another way of doing this already, perhaps having a delta_override option on the throttle_average filter would work? Where whenever the input value is over that delta, it clears the average history for the current window and immediately sends an update with the input value, then goes back to averaging.

Additional context I flashed an Emporia Vue 2 energy monitor with ESPHome and the custom component available for that, and I'm trying to reduce the update frequency of power sensors overall while still having instant updates when meaningful deltas happen.

A percentage delta that has a minimum value would be lovely. I can implement that kind of delta with a lambda, but if it ends up being built into throttle_average then the flexibility would be lost. The importance of this is because a 5% difference on 15 watts isn't meaningful, and 5 watts difference on 2500 watts also isn't meaningful (enough to warrant an instant update anyway), so having a minimum on that percentage delta is important.

[ColColonCleaner]

This is the lambda I'm using for the 'percentage delta with minimum' part of it. But getting this worked into a throttled average is the goal.

- lambda: |-
   static float transition_threshold = 75.0;
   static float delta_percentage = 0.10;
   static float delta_minimum = 5.0;
   static float last_value = NAN;
   if (isnan(last_value) || x == 0)
     return last_value = x;
   float x_delta = std::abs(x - last_value);
   if (x_delta >= delta_minimum && (x < transition_threshold || std::abs(x_delta / x) > delta_percentage))
     return last_value = x;
   else
     return {};

[nagyrobi]

Shouldn't a global variable be used with this lambda to hold the previous value, to compare to?

[ColColonCleaner]

@nagyrobi Static variables persist between runs of the lambda, so last_value will be the last value after the first run. Using a separate global variable for each sensor using the filter is also a solution but that is much more cumbersome with the amount of configs required if many sensors use it.

[nagyrobi]

Static variables persist between runs of the lambda

Really? I wasn't aware of that. Is this in the doc somewhere?

[ColColonCleaner]

Really? I wasn't aware of that. Is this in the doc somewhere?

Yes, in this section. Look under 'tip' at the bottom. Global variables are still useful if you want the value to be accessible outside of the lambda, but if you only care about it inside, then a static variable is sufficient and much simpler (especially if you have a template lambda that is used by many sensors).

https://esphome.io/guides/automations.html#templates-lambdas

Hopefully someone has a solution to my original problem though. Been trying to figure out the proper way to call set_interval from a lambda (to build my own throttle_average) and provide a function for it, if that's even possible without screwing things up.

[nagyrobi]

Maybe in new https://github.com/esphome/esphome/pull/5423

[ColColonCleaner]

Looking through that PR and the related issue i don't think that solves this particular issue. Unless it's able to kill/restart the throttle_average filter on a particular sensor.

[ColColonCleaner]

Would be wonderful to have this looked into. Is there any additional information I should provide?

[ColColonCleaner]

Would love to provide any additional info or insight that's needed to have this looked at. Even if it's not implemented right away, someone chiming in on whether it's a viable feature would be great.

[BartRuSec]

Ok, some time ago I had the similar problem and found solution using lambda filter only (mix of delta & throttle average)

deltaThrottle.lambda:

static float last=0;
static int count=-1;
static float sum=0;
count++;
if (count==0) return x;
if (fabs(last-x)>=${delta}) {
    last=x;
    count=sum=0;
    return x;
} else if (count>=${window}) {
    float result=(sum+x)/count;
    count=sum=0;
    return result;
} else {
    sum+=x;
    return {};
}

than in yaml you can use it as:

 filters:
        - lambda: !include { file: deltaThrottle.lambda, vars:{delta:  'id(delta_adjustment).state',window: 'id(throttle_window).state'}}

The downside of this approach, is that when delta happens you are loosing the last throttle average value however I found it pretty useful to detect peaks.

[AJolly]

I'd like to figure this out too!

[ssieb]

The original request doesn't really make sense. What are you expecting the result to look like? You are doing an average, so any large changes will be gradually applied. But now you want to output a large value directly. So you're going to get that second graph where the large value gets output, but then the next value isn't that much larger than the previous one, so the average gets sent again. And unless there's another large jump, the average will continue to be published. So what exactly do you want to happen?

[ColColonCleaner]

@ssieb here's an example. You have a sensor for power usage. You want this sensor to be quick to update, but you don't want it to spam HA with requests if there is a lot of small fluctuation. Instant updates for large changes, but throttling small fluctuations to every 10 seconds to avoid data spam.

We can't use 'throttle' because that filter discards updates which fall inside its window. So this scenario fails: A 1000W resistive load activates, a heater fan for example. The raw sensor reads 0W -> 15W -> 1000W in quick succession. The 1000W update is dropped because its inside the throttle window. The sensor remains at 15W.

We can't resolve this with a 'heartbeat' filter because that means all 18 breakers are spamming data every few seconds.

Throttled moving average seems like a good solution to the data spam, and it doesn't drop the last update like 'throttle' does. Unfortunately this filter makes instant updates for large deltas impossible. If the power usage jumps from 0W to 1000W I want automations to handle that immediately, not waiting for the throttled average window to finish.

Expected behavior: throttled average sensor behaves as normal unless a new value comes in that triggers the delta. If the delta is triggered the average history is nuked and starts fresh again at the new sensor value. Example: delta set at 50W. Reading is hovering around 30W, sensor uses throttled average as normal, slowly updating. Reading jumps to 60W, sensor uses throttled average as normal, slowly updating. Reading jumps to 1200W, sensor immediately jumps to 1200 and starts a fresh average window from there.

[ssieb]

Wouldn't the regular throttle or'ed with the delta work?

[nagyrobi]

@ColColonCleaner your approach would result in errorneous meaaurements on long term. For the use case you specified, i'd use a copy sensor, and apply moving average filter for storing the data, and present that to HA as the sensor. I'd set the original sensor to internal, and use necessary filters to detect the sudden changes, and trigger an event in HA (or a binary sensor change) to run the automations as desired.

So split the goals according to the needs.

[ColColonCleaner]

@ssieb I could absolutely do that IF i knew it was not discarding the latest data point. I.E. If the throttle filter kept track of the last value it received and published that value when the throttle window expires.

Reasoning: If a small change happens (one that does not trigger the delta OR), say power usage goes from 20W -> 19W -> 12W within the throttle window. That 12W data point is dropped. This means the sensor is stuck showing 19W instead of 12W. HA doesn't know the value has dropped, and I'm also calculating energy usage on the wrong value. 8W of fluctuation is a lot for a 20W load, but 8W of fluctuation for a 1200W load is noise.

[ssieb]

Do you understand how the throttle filter works? Any value published within the filter time is dropped. The next value after that is published. It doesn't hold on to any values. Or are you saying that only the 12 was in the window? Your description isn't clear. But as long as the sensor is updating, there will be a value.