Rieman integration looses source attribute

[erkr]

The problem

I configured a riemann integration sensor in yaml. A crucial part is the source attribute. After a while I noticed in the developer tools that attribute got lost, and the state became unavailable. A restart doesn't bring that attribute back!

What version of Home Assistant Core has the issue?

Core-2023.5.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Riemann integration

Link to integration documentation on our website

https://www.home-assistant.io/integrations/integration/

Diagnostics information

No response

Example YAML snippet

  - platform: integration
    source: sensor.slaapkamer_airco_energy_current
    name: Slaapkamer airco energy summation
    unit_prefix: k
    unit_time: h
    round: 2

Anything in the logs that might be useful for us?

No response

Additional information

This is what I see in the developer tool. Initially the was a source attribute

state_class: total
unit_of_measurement: kWh
device_class: energy
icon: mdi:chart-histogram
friendly_name: Slaapkamer airco energy summation

[home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (integration) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `integration` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign integration` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

integration documentation integration source _{^{(message by IssueLinks)}}

[dgomes]

Any logs ? Please turn on debugging for this component

[erkr]

Nothing was logged (at warnings level). I noticed the same when I create a sensor with the UI setup. The only difference is that the sensor then becomes a helper, but it also looses the source attribute after a while.

I will enable debug logging

[erkr]

Update: I added this for debug log:

logger:
  default: warning
  logs:
    integration: debug
    homeassistant.components.integration: debug

Nothing in the log though!!

The existing sensor simply doesn't get the source attribute any more. Even not after deleting rebooting and re-adding and rebooting. That is weird!

So I copied the sensor configuration with just another name (added 2) and than the source attribute is there in the new sensor only. The configuration:

  - platform: integration
    source: sensor.slaapkamer_airco_energy_current
    name: Slaapkamer airco energy summation
    unit_prefix: k
    unit_time: h
    round: 2
  - platform: integration
    source: sensor.slaapkamer_airco_energy_current
    name: Slaapkamer airco energy summation 2
    unit_prefix: k
    unit_time: h
    round: 2

First sensor in development tools:

state_class: total
unit_of_measurement: kWh
device_class: energy
icon: mdi:chart-histogram
friendly_name: Slaapkamer airco energy summation

New sensor in development tools:

state_class: total
source: sensor.slaapkamer_airco_energy_current
icon: mdi:chart-histogram
friendly_name: Slaapkamer airco energy summation 2

Please note that device class is missing for the new one!!!!

[erkr]

Because the input sensor is stable at 5 watts, I forced a reload of the source sensor with this service call:

service: homeassistant.reload_config_entry
data: {}
target:
  entity_id: sensor.slaapkamer_airco_energy_current

After that the new sensor changed and the device class is added and the source is gone (same as the first one) Now there where loggings:

2023-07-19 11:54:27.931 DEBUG (MainThread) [homeassistant.components.integration.sensor] Could not apply method trapezoidal to 5 -> unavailable
2023-07-19 11:54:27.932 DEBUG (MainThread) [homeassistant.components.integration.sensor] Could not apply method trapezoidal to 5 -> unavailable
2023-07-19 11:54:33.522 DEBUG (MainThread) [homeassistant.components.integration.sensor] Could not apply method trapezoidal to unavailable -> 5
2023-07-19 11:54:33.523 DEBUG (MainThread) [homeassistant.components.integration.sensor] Could not apply method trapezoidal to unavailable -> 5

[erkr]

Changed the method to left. Still some issues in logs:

2023-07-19 12:05:57.215 DEBUG (MainThread) [homeassistant.components.integration.sensor] area = 296.8136850000000137583811011, integral = 0.00008244824583333333715510586142 state = None
2023-07-19 12:06:00.548 DEBUG (MainThread) [homeassistant.components.integration.sensor] Could not apply method left to unavailable -> 5

[erkr]

Hope this provides the required information

[dgomes]

Some properties are are only created when the source first publishes.

As to why the "source" property disappears I don't have any hint... did it work in previous releases ?

The debug messages are normal warning, due to a unreliable source that often becomes unavailable....

[erkr]

Hi, don't know if it worked before. First time I use it, and it doesn't work. The unaffordable is not because of the sensor. That is an airco power sensor that is idling at 5 watts, but I don't get any statistics: Therefore in tried to reload the source entity config, to see if that triggers the statistics. But no, the reload is probably what causes the unavailable in the logging.

[dgomes]

if the source stays idle (no change in state) then Riemann will no work, because it only updates when the source updates.

[erkr]

It's not idle. It's a stable 5 watts because the airco is on standby. Please trigger it at least once per time_unit with a timer. Home assistant is optimising to only update sensors for state changes. Artificially updating these source sensors in order to support integration violates that concept and increases the recorder database. I noticed the same issue with the derivative sensor.

Best Eric

[dgomes]

stable = idle from the view point of home assistant internal state machine. You are right on the reasons for that, but unfortunately it is also the reason you don't get updates on Riemanm or Derivative sensor.

[erkr]

Ok idle🤓. From a user perspective it is very weird that sensors during stable periods won't be integrated (time * state) nor derive (zero).

[erkr]

Just for those running into the same issue that the derivative and integration sensors don't work for "idle" source sensors, a work around!

As long the author doesn't add some extra timer, extra triggers can be added by creating an extra template sensor. This template sensor updates an attribute periodically (found that solutions somewhere else but don't recall the source). So the value doesn't need to change. I named the template sensor 'nohist*' to exclude them with a filter in the recorder!

My definitions in 'configuration.yaml':

sensor calculated: 

  - platform: derivative
    source: sensor.no_hist_total_gas_usage
    name: gasverbruik 
    unit_time: h
    time_window: "00:05:00"
    unit: m3/uur
    round: 3

  - platform: integration
    source: sensor.no_hist_slaapkamer_airco_energy_current
    name: Slaapkamer airco energy summation
    unit_prefix: k
    unit_time: h
    method: left
    round: 2

The two extra template sensors:

- trigger:
  - platform: time_pattern
    minutes: "/2" 
  - platform: state
    entity_id: 
     - sensor.total_gas_usage
    not_from:
      - "unknown"
      - "unavailable"
    not_to:
      - "unknown"
      - "unavailable"
  sensor:    
  - unique_id: no_hist_total_gas_usage 
    name: no_hist_total_gas_usage
    availability: "{{ has_value('sensor.total_gas_usage') }}"
    state: "{{ states('sensor.total_gas_usage') }}"
    attributes:
      dummy: "{{ now().minute }}"
    unit_of_measurement: m³
    device_class: gas

- trigger:
  - platform: time_pattern
    minutes: "/10" 
  - platform: state
    entity_id: 
     - sensor.slaapkamer_airco_energy_current
    not_from:
      - "unknown"
      - "unavailable"
    not_to:
      - "unknown"
      - "unavailable"
  sensor:    
  - unique_id: no_hist_slaapkamer_airco_energy_current 
    name: no_hist_slaapkamer_airco_energy_current
    state: "{{ states('sensor.slaapkamer_airco_energy_current')  }}"
    availability: "{{ has_value('sensor.slaapkamer_airco_energy_current') }}"
    attributes:
      dummy: "{{ now().minute }}"
    unit_of_measurement: W
    device_class: power

Note: for integration the frequency probably doesn't need to be very high to get e.g usage per hour statistics.

And finally the option to add an exclusion filter in the recorder:

recorder:
  db_url: !secret recorder_db_url
  purge_keep_days: 4
  exclude:
    entity_globs:
    - sensor.no_hist_*

[erkr]

@dgomes Since I made the template sensor in between, the source attribute doesn't disappear anymore. I think that happens when the source sensor gets unavailable (e.g. that happens shortly when you call the service to reload the source sensor config). Guess that will be reproducible.

[dgomes]

I've created an alternative to the templates: https://github.com/dgomes/ha_sampler

[erkr]

The point is that it should not be needed to create extra helpers or templates, just to handle stable sensors. IMO these are workarounds, for an issue that should be fixed. You just need to add a timer to the integrations to trigger in case the values are stable. And you can optionally restart the timer each time the sensor updates. The solution remains event driven this way.

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[erkr]

Bump

[solazs]

Yeah, bump. Reloading my template sensor configuration means all the natural gas related UI items are unavailable until the furnance kicks in.

[erkr]

Try to make the template sensor trigger based. Those will restore their values

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[erkr]

Bump

[erkr]

I noticed there is as PR for triggering calculations when the sensor is stable. With that fix attempts like calling reload the source entity (the reload was causing the loss of the source attribute) don't make sense anymore. I close this issue.