Riemann sum integral sensor not updating while source sensor value is stable

[itssimon]

The problem

The Riemann sum integral sensor currently only updates when the source sensor updates, which is not always sufficient. Ideally it would calculate a new value periodically too, even if the source sensor value remains stable.

For example: I’m monitoring the energy used by our pool filtration system. It’s power usage stays exactly the same throughout the day after a brief priming phase. In this scenario the integration sensor does not calculate the energy used until the end of the day when the pump stops.

What version of Home Assistant Core has the issue?

2023.2.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Integration - Riemann sum integral

Link to integration documentation on our website

https://www.home-assistant.io/integrations/integration/

Diagnostics information

No response

Example YAML snippet

sensor:
  - platform: integration
    source: sensor.pool_filter_pump_power
    name: Pool Filter Pump Energy
    unit_prefix: k
    round: 2
    method: left

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (integration) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `integration` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign integration` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

integration documentation integration source _{^{(message by IssueLinks)}}

[rgb-Argentum]

I'm afraid that this is not a 'feature' since integration is a function of values and time. So excluding time completely and keeping the same value until the source sensor value changes means that this integration is only half-implemented. I've tried forcing update of the base sensor and the integral itself through an automation but it didn't change anything. The only 'solution' is to insert some fakes into the sensor value stream time to time. But that makes the original sensor history look ugly.

[dgomes]

The current implementation is event driven to support sensors that update often (which is also when Riemann Sum best approaches the integral function). You are requesting a new feature because it requires a completely new implementation that is based on periodic polling of the source sensor. It's not that it can't be done, it's just that it isn't a bug...

[rgb-Argentum]

I know that the current implementation is based only on sensor update. But that is not exactly true: HASS doesn't broadcast any updates if the new value of the sensor is the same as the previous. That means the current implementation is practically changes only when the sensor value changes and not when the updates received from the sensor device. So the integration should also include an (optional, configurable) timer that periodically updates the integration value based on the last value and the time elapsed since that moment. There is no need to check or poll the sensor.

[kaijk]

My rain gauge sends a new reading every 10 minutes with 0.01 inch resolution. When its raining, successive values are often the same: 0.01, 0.01, 0.02,. 0.02, etc. This wreaked havoc with HA. It missed all the new, valid, values that happened to be the same value as the last value. Each new value is cumulative, not a repeat of the current state if unchanged from the prior. I solved this by adding a small +/- random lambda value to each incoming data point.

This doesn't guarantee that two identical 10 minute readings will not also get identical random adders, but, good enough for me. So, until another solution arises, here's mine.

The command assures that the new data are no more than a minute old so stale data does not get interpreted as new.

In the if else block. I add both positive and negative lambdas to even it out around zero over time, and never add a lambda when the new reading is zero:

- platform: command_line
  name: 'Acurite 10-Min Rainfall'
  command: "[[ $((`date +%s` - `date +%s -r '/config/externaldata/acuriteweatherdata.txt'`)) -le 60 ]] && cat /config/externaldata/acuriteweatherdata.txt || echo '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.0' "
  # Add a small lambda to try to keep successive values unique so the Utility Meter will add each value to the sum.
  value_template:  >-
    {% set val = ( value.split( ',' )[14] | float(0.0) ) | round(2) %}
    {% if val == 0.0 %}
       0.0
    {% else %}
       {{ ( val  + ( [ 0.0, -0.000001, 0.000001, -0.000002, 0.000002, -0.000003, 0.000003, -0.000004, 0.000004, -0.000005, 0.000005, -0.000006, 0.000006, -0.000007, 0.000007, -0.000008, 0.000008, -0.000009, 0.000009, -0.000010, 0.000010,-0.000011, 0.000011, -0.000012, 0.000012, -0.000013, 0.000013, -0.000014, 0.000014, -0.000015, 0.000015  ] | random ) ) | round(6)  }}
    {% endif %}
  unit_of_measurement: 'in'

[dgomes]

To possibly address the lack of updates in some sources, I've created a Helper integration: https://github.com/dgomes/ha_sampler

If you run the source through this custom helper, and then use this custom helper as the source of Riemann Integration, you should get better results.

PLEASE: don't create issues or reply to this thread about this custom_component, use the GitHub repository of the custom_component!

[thecem]

So what is the solution to fix this? to add a helper for a helper?

[dgomes]

The sampler helper is a stop gap for some use-cases.

I don't even know the real impact.

If any of you is testing/using, feedback would be nice.

[kaijk]

I don't see ha_sampler in either Integrations or HACS. I'm on 2023.4, though, if it was included later

On Mon, Jun 5, 2023, 12:00 PM Diogo Gomes @.***> wrote:

The sampler helper is a stop gap for some use-cases.

I don't even know the real impact.

If any of you is testing/using, feedback would be nice.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/88940#issuecomment-1577308461, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL2THYV4SDXVL36S2OKECNLXJYUFZANCNFSM6AAAAAAVLL4TF4 . You are receiving this because you commented.Message ID: @.***>

[dgomes]

it's a trial, it's not part of HACS (although you can use HACS to install it)

[asimar75]

Work around, suggest to create automation

alias: reload INTEG description: reload INTEG trigger:

• platform: time_pattern minutes: "1" condition:
• condition: state entity_id: sensor.ABC state: unavailable
• condition: or conditions: []
• condition: state entity_id: sensor.XYZ state: unavailable action:
• service: homeassistant.reload_config_entry data: {} target: entity_id:
  • sensor.ABC
  • sensor.XYZ mode: single

[kaijk]

@dgomes I'm out for the next month or so and have not been able to test your idea. Sorry, and I really appreciate your digging into this issue.

HA's assuming successive identical sensor readings reflect no-change in the state is a large and lurking problem.

Rain Gauges are a great test case in which successive 10-minute intervals each report, say, 0.01 in of new rain. HA without a fix or workaround ignores those successive identical new incremental readings by assuming the readings are not incremental, but cumulative to date.

[erkr]

My two cents, read the values at least ones per unit_time. Just add a simple timer that will give an additional event when unit_time expired. I predict that having these extra events won’t harm and don’t require a completely new implementation. I suspect it will even simplify your current design! Now you somehow have to chop integrations results over various unit time buckets, when the next sensor event took longer than the unit time.

Note: the derivative integration has a very similar issue. When the sensor stops changing, the derivative shall become zero, but doesn't without further sensor updates

[Steve2017]

This looks like it could be the same problem I have with Riemann sum sensors which become unavailable when a source sensor becomes unavailable for a short period while it is producing unchanged data.

I have the Enphase integration that supplies data about electricity consumption and solar production. For some unknown reason the device goes off-line for a minute at 23:00, leaving all consumption and production sensors unavailable for that minute. At that time the solar production is zero and has been for several hours.

Template sensors using the integration sensors (Solar Export Power & Grid Import Power) come back after showing unavailable for the same minute as the integration sensors. This is the solar export power sensor:

        state: >
          {{ [0, (states('sensor.envoy_current_power_production') | int - 
                states('sensor.envoy_current_power_consumption') | int)  /1000] | max }}

The Grid Import sensor resumes showing my consumption as expected (about 500-watts). At this time of night, the solar export sensor is showing 0kW before and after the event, but is available before and after the outage.

Downstream from that I have a Solar Export Energy sensor using Reimann Sum:

  - platform: integration
    name: Solar Export Energy
    source: sensor.solar_export_power 
    unit_time: h
    method: left

This sensor and all sensors relying on it as a source (daily energy export, $ earned from the export, net cost of power etc.) become unavailable and remain that way until solar production resumes the next morning. Even though sensor.solar_export_power has returned to available, the Reimann Sum sensor is unavailable until sunrise.

The downstream Reimann Sum sensors using the Grid Import sensor have no such issue, possibly because they are receiving changijng data?

I thought the "available" template command might help, but I am not sure how to use that.

BTW I thought this only started happening in May - ie: 2023.5.0

[erkr]

Hi, As long the author doesn't recognise stable sensors as a valid use case, I created extra template sensors as a work around to add time triggers in case the value remains stable. See https://github.com/home-assistant/core/issues/96875#issuecomment-1656810325

[erkr]

@Steve2017 the other issue was related to loosing the source sensor when it shortly became unavailable. Maybe good to know that seems to be solved as well since I use the template workaround. Maybe related to your issue

[Steve2017]

since I use the template workaround.

It could be - but we shouldn't need a workaround.

I am trying to understand your solution because I still don't have a good understanding of the functions.

[erkr]

The integrate and derive integrations only trigger calculations when the source sensor updates. With the time pattern, the template sensor periodically changes the content of a dummy attribute, which also counts as an update, without changing the sensor value. So that triggers a new calculation for stable sensors

[AleXSR700]

It's not that it can't be done, it's just that it isn't a bug...

I missed this thread during my search and had opened a parallel issue here https://github.com/home-assistant/core/issues/102057

I also explained why it is a bug. Essentially, the current Rieman sum reports 0 kWh consumption even if the source constantly consumes 1 kWh. So the result of the integration is wrong. If no update is performed, the returned value must not be 0. Returning 0 is simply false. An hourly sum must either force an update even if the input data is constant or must return unknown/unavailable.

By the way, one could argue that the integration of a constant value is the simplest form of an integration ;-)

[Steve2017]

I have moved away from using this Riemann Sum sensor because it is unreliable. That rules out using the Enphase integration because it needs Riemann Sum for some data. I have moved to REST for my power/energy data and it is working without issue.

In my case the constant reporting (0kW) was not the problem. The system went off-line for 60 seconds and that caused the sensor to fail until it had a reading other than 0kW.

I found the comments that the issue with the Riemann Sum sensor ‘isn’t a bug but a feature request’, to be strange.

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[erkr]

bump

[ronweikamp]

I have made an attempt to solve this issue https://github.com/home-assistant/core/pull/110685

[erkr]

This looks exactly as the solution we need. Finally get rid of helpers. Thanks! Can't wait to have it into the release

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[erkr]

Bump

[Steve2017]

With a change of address I have had to go back to Riemann Sum for energy sensors and I am still experiencing the problems list above. The child sensor becomes unavailable because the source sensor goes offline for a minute and when it does come back it is with unchanged data.

[dgomes]

https://github.com/home-assistant/core/pull/110685 should fix this issue

[kaijk]

😃 I don't know how to tell, but will this make 2024.6 that hit beta yesterday? (I'd love to know how to tell)...

Thanks, all, for the diligence and maths in making this happen.

On Thu, May 30, 2024 at 11:40 AM Erik Montnemery @.***> wrote:

Closed #88940 https://github.com/home-assistant/core/issues/88940 as completed via #110685 https://github.com/home-assistant/core/pull/110685 .

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/88940#event-12990318355, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL2THYXSYJGNEHZDEM4PAT3ZE5XDBAVCNFSM6AAAAAAVLL4TF6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSHE4TAMZRHAZTKNI . You are receiving this because you commented.Message ID: @.***>

[AleXSR700]

I think once it is merged into dev it is automatically part of the next build. So if I am not mistaken it should either be in 2024.6 or in 2024.6 update 1.

[Settl3r]

I think this is not yet implemented in the current release 2024.6.2: Invalid config for 'integration' from integration 'sensor' at configuration.yaml, line 420: 'max_sub_interval' is an invalid option for 'sensor.integration', check: max_sub_interval

Or have I overlooked something?

[erkr]

I’m lost as well, which release will provide us this improvement?

[dgomes]

It's a new feature, will get into 2024.7.0

[AleXSR700]

Is the max_sub_interval every minute on the minute or every minute after 60 seconds?

So, if you set an update interval of 60 min, will it be every hour on the hour or will it be 60 min after reboot/yaml reloading (e.g. 07:03 -> 08:03)?

[rgb-Argentum]

@AleXSR700: The timer starts after the last received sensor value change. So if the sensor value is constant then it will pump the integration every max_sub_interval minutes after restart of HA.

[AleXSR700]

Oh, okay, so I might remain better of forcing a brief change to the source rather than using the max interval. Because the max intervall could delay the hourly update by 59 minutes if you select 60 min intervals.

[erkr]

It works, thank!!!!! By the way, documentation could give an example how to define the time.

[AleXSR700]

By the way, documentation could give an example how to define the time.

It does.

[erkr]

By the way, documentation could give an example how to define the time.

It does.

This is the documentation: Max sub-interval Applies time-based integration if the source did not change for this duration. This implies that at least every max sub-interval, the integral is updated. If you don’t want time-based updates, enter 0.

My point was, what to configure. Seconds? Or some time format (hours, minutes, seconds). Is there some range limit.... But I overlooked this:

    max_sub_interval:
      minutes: 5

Remains the range question 😉

[kaijk]

See example here. https://www.home-assistant.io/integrations/integration/#energy What I don't understand is if the max_sub_interval: is omitted, does it default to "zero," or to some interval? The documentation says to specify "0" if you don't want it set, but is silent on what happens if no value is provided.

On Tue, Jul 9, 2024 at 9:48 AM Eric @.***> wrote:

By the way, documentation could give an example how to define the time.

It does.

This is the documentation:

Max sub-interval Applies time-based integration if the source did not change for this duration. This implies that at least every max sub-interval, the integral is updated. If you don’t want time-based updates, enter 0.

My point is, what to configure. Seconds? Or some time format (hours, minutes, seconds). Is there some range limit....

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/88940#issuecomment-2218213335, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL2THYU5VJR26SSTULVVC2DZLQH6LAVCNFSM6AAAAAAVLL4TF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYGIYTGMZTGU . You are receiving this because you commented.Message ID: @.***>

[erkr]

And how to specify the zero. Like this:

    max_sub_interval:
      minutes: 0

Or

    max_sub_interval: O

[AleXSR700]

But I overlooked this:

    max_sub_interval:
      minutes: 5

Remains the range question 😉

That's the one ;-)

What kind of range restriction? Why would there be one? And why would anybody define a range that might be so high that it could cause a range issue? :-D

[AleXSR700]

The documentation says to specify "0" if you don't want it set, but is silent on what happens if no value is provided.

It is optional, so by default it is disabled, i.e. 0

        self._max_sub_interval: timedelta | None = ( 
             None  # disable time based integration 
             if max_sub_interval is None or max_sub_interval.total_seconds() == 0 
             else max_sub_interval

[rgb-Argentum]

You could add this sensor from Lovelace (and now we could finally edit all parameters!): Go to Settings/Devices & Services/Helpers tab/Create Helper button/Integral sensor. This is the way that the manual suggests because the time interval is zeroed out by default (0:00:00 h:m:s).

To edit the sensor find it on the Helpers tab mentioned before or simply press 'e' on the keyboard (or tap on the magnifier icon on the upper right corner) and type the integral sensor name. Then click on the sensor, then the cogwheel on the up-right corner and select Integral sensor options item. You could change the source sensor, integration method, precision and the time interval.

[AleXSR700]

Who was that recommendation for? :)

I think there are still a bunch of people who "like" using yaml because it gives more control to the user, especially when it comes to setting up a new system or restoring data.

[frenck]

Alright, this issue has been fixed and closed. The current conversation going on here, seems more fitting for our community forums or Discord chat.

Let's continue the conversation there 👍

../Frenck