Sensor groups sum is sometimes wrong

[seblu]

The problem

Sensor groups used to sum values are computing incorrect (lower) values. This is happening sometimes when HA restart.

Here is an example of 3 outlets providing their total energy consumption and a sum sensor group. The group has been created with the helpers frontend with ignore_non_numeric set to false.

As you see, the blues lines should not have dropped.

Here is the core.config_entries

{
  "entry_id": "63b5f8eb1ad7b8c327baf1c4847453b5",
  "version": 1,
  "domain": "group",
  "title": "Room3 Energy",
  "data": {},
  "options": {
    "group_type": "sensor",
    "name": "Room3 Energy",
    "entities": [
      "sensor.room3_outlet_ne_energy",
      "sensor.room3_outlet_swl_energy",
      "sensor.room3_outlet_swr_energy"
    ],
    "hide_members": false,
    "type": "sum",
    "ignore_non_numeric": false
  },
  "pref_disable_new_entities": false,
  "pref_disable_polling": false,
  "source": "user",
  "unique_id": null,
  "disabled_by": null
}

When the sensor group state is used by an Utility Meter, it is interpreted as a reset. Then, the meter becomes crazy.

What version of Home Assistant Core has the issue?

2023.10.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

group

Link to integration documentation on our website

https://www.home-assistant.io/integrations/group/#sensor-groups

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (group) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `group` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign group` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

group documentation group source _{^{(message by IssueLinks)}}

[home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (utility_meter) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `utility_meter` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign utility_meter` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

utility_meter documentation utility_meter source _{^{(message by IssueLinks)}}

[seblu]

Here is the previous graph with the addition of:

• an utility meter (in yellow) computing the monthly consumption based on the sensor group energy (Room3 Energy). You can better see the chain of problems this creates.
• the HA uptime sensor. Looks like it's not only happening when HA restarts.

[seblu]

I have log lines for the 23:45 unavailability of the members of the group. There is no drop in the group sensor as the 3 sensors are all unavailable at the same time.

2023-10-31 23:45:49.922 WARNING (MainThread) [homeassistant.components.group.sensor] Unable to use state. Only numerical states are supported, entity sensor.room3_outlet_ne_energy with value unknown excluded from calculation
2023-10-31 23:45:49.922 WARNING (MainThread) [homeassistant.components.group.sensor] Unable to use state. Only numerical states are supported, entity sensor.room3_outlet_swl_energy with value unknown excluded from calculation
2023-10-31 23:45:49.922 WARNING (MainThread) [homeassistant.components.group.sensor] Unable to use state. Only numerical states are supported, entity sensor.room3_outlet_swr_energy with value unknown excluded from calculation

I did find an older example with another sensor group drop with the associated log line. This time, only one member is missing. Sorry, it's not the same sensor group and members, but I have the log lines.

2023-10-28 13:53:58.815 WARNING (MainThread) [homeassistant.components.group.sensor] Unable to use state. Only numerical states are supported, entity sensor.lounge_outlet_dishwasher_energy with value unavailable excluded from calculation

Expected behavior from the doc:

If ignore_non_numeric is false then group state will be unavailable if one member is unavailable or does not have a numeric state.

My guess is that for some reason, the sensor group is excluding members for computing the value instead returning unavailable. This is why we are seeing a decrease, and when all the sensors are together excluded there is no drop in the sensor group.

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[seblu]

bump

[Gizmo0815]

I have the same issue. I sum up the energy total of two fridges in a group helper. The group helper value (dark blue in the screenshot) ocassionally drops from the sum of both entities to the value of one entity (purple in the screenshot) and then goes back to the sum of both.

I use the group helper in my energy dashboard and this behavior results in displaying an energy consumption as high as the energy total of entity two (light blue in the screenhot) for the current day(!).

Is there an option to prevent the value of the group from decreasing?

[Xygen]

Same issue here. A Group Sum helper sensor seems to be reset temporarily during reboot:

Which results in unrealistically high consumptions in the energy dashboard:

Helper Sensor is "state_class: total_increasing"

Is there a way to fix this?

[frli4797]

bump

[Xygen]

I'm still having this issue daily. I thought it might be the one of the Shelly Plugs became unavailable temporarily... But I don't know :(

[mschaeuble]

@Xygen Could you solve the problem in the meantime? I have a similar problem: My group sum sensor consisting of 3 shelly devices has irregular drops (always at 02:01 AM):

In the HA-logs, I could find the following entries at the time the drops occur: Error fetching shellyem3-40F52000B9A4 data: Error fetching data: DeviceConnectionError()

EDIT: It looks like my problem is related to https://github.com/home-assistant/core/issues/119994

[Xygen]

@Xygen Could you solve the problem in the meantime? I have a similar problem: My group sum sensor consisting of 3 shelly devices has irregular drops (always at 02:01 AM):

In the HA-logs, I could find the following entries at the time the drops occur: Error fetching shellyem3-40F52000B9A4 data: Error fetching data: DeviceConnectionError()

EDIT: It looks like my problem is related to #119994

Unfortunately no… I‘m still fixing it manually via dev tools on a daily basis :(

[IWR69]

I'm having exactly the same problem as the author of this post, well almost - my invalid values are much higher that they should be. I have also noticed, after reading a comment above, that these invalid values are written to the statistics_short_term table immediately after a Home Assistant restart.

I originally though it was a frontend issue but with the help of karwosts we have determined that it is most likely a core issue with the recorder integration. Here is my original post https://github.com/home-assistant/frontend/issues/22246 although I will describe the problem again now that I have had help from karwosts to identify the problem more accurately.

I have a dashboard chart that is displaying invalid humidity values for one of my sensors.

This is the yaml for the graph...

chart_type: line
period: 5minute
type: statistics-graph
entities:
  - entity: sensor.bedroom_humidity
    name: Bedroom
  - entity: sensor.living_room_humidity
    name: Lounge
  - entity: sensor.office_humidity
    name: Office
days_to_show: 2
hide_legend: false

The Living Room Humidity (Lounge) sensor is actually a Group Helper that calculates the arithmetic mean from two humidity sensors ignoring non numeric data. One of these sensors has been unavailable for some time but the other sensor is working fine and recording only valid humidity values i.e. < 100, in the statistics_short_term table. Below I have queried the table for this working sensor and the 3 timestamps when the odd values appeared for the group helper...

As you can see the values are all good.

However when I queried this table for the Group Helpers values I found these 3 invalid values and their timestamps corresponded with the spikes in the graph above. Not only is the mean value crazy but the max value is also 100 times the min value.

I also visually checked the whole table, for this metadata_id, for any mean value that looked odd but these 3 were the only odd values I could see.

Finally when reading this post someone mentioned that the problem appears to happen when restarting Home Assistant. I also noticed this after restarting Home Assistant in safe mode for my previous post. Another spike appear on the graph but I didn't realise at the time that it was caused by the restart. Anyway after reading this I did another restart to test this and again another spike appeared on my graph. Querying statistics_short_term I can see a 4th invalid record (high mean value) for the group header and a good record with valid values for Nest_Thermostat_Humidity sensor. Clearly there appears to be a problem with Group Helpers, where one of the sensors is unavailable and ignore non numeric is enabled, and Home Assistant is restarted.

[IWR69]

Do I need to repost my comment above as I notice no one is assigned to or commented on this issue? It feels like I wasted a lot of time and effort posting my comment.

[dgomes]

Although I've unassigned myself I can tell that the core team is looking into this and similar issues.

The issue lies in the unavailable and unknown states of any of the group members.

[Xygen]

Thank you for the update, @dgomes!

Is there another issue we can follow for status updates? I'm eager to help in any way to resolve this.

As a potential workaround, could we implement an option to prevent the value of a total-increasing sensor from decreasing? This could be set on a per-sensor basis.

Thank you!

[Xygen]

Hello! What do you think of the idea of decoupling a sensor that sums up 2 or more source sensors by only adding positive changes to the target sensor?

In my example below, the previous values are stored as attributes in the target sensor, compared to the new values, and only if there's a positive change in the source sensors, the difference is added to the value of the target sensor. This has been working flawlessly for two weeks now.

I'd be happy if I could offer the community a useful workaround.

template:
  - sensor:
      - name: "Freezer Total Energy"
        unique_id: "freezer_total_energy"
        state: >
          {% set input1 = states('sensor.s3_freezer_kitchen_energy') | float(0) %}
          {% set input2 = states('sensor.s9_freezer_basement_energy') | float(0) %}
          {% set prev_input1 = state_attr('sensor.freezer_total_energy', 'previous_input1') | float(0) %}
          {% set prev_input2 = state_attr('sensor.freezer_total_energy', 'previous_input2') | float(0) %}
          {% if input1 == 'unavailable' or input2 == 'unavailable' %}
            {{ states('sensor.freezer_total_energy') | float(0) }}
          {% else %}
            {% set change_input1 = max(input1 - prev_input1, 0) %}
            {% set change_input2 = max(input2 - prev_input2, 0) %}
            {{ states('sensor.freezer_total_energy') | float(0) + change_input1 + change_input2 }}
          {% endif %}
        availability: >
          {{ states('sensor.s3_freezer_kitchen_energy') not in ['unknown', 'unavailable'] and states('sensor.s9_freezer_basement_energy') not in ['unknown', 'unavailable'] }}
        attributes:
          state_class: "total_increasing"
          unit_of_measurement: "kWh"
          device_class: "energy"
          icon: "mdi:lightning-bolt-circle"
          friendly_name: "Freezer Total Energy"
          previous_input1: >
            {% if states('sensor.s3_freezer_kitchen_energy') not in ['unavailable', 'unknown'] %}
              {{ states('sensor.s3_freezer_kitchen_energy') }}
            {% else %}
              {{ state_attr('sensor.freezer_total_energy', 'previous_input1') }}
            {% endif %}
          previous_input2: >
            {% if states('sensor.s9_freezer_basement_energy') not in ['unavailable', 'unknown'] %}
              {{ states('sensor.s9_freezer_basement_energy') }}
            {% else %}
              {{ state_attr('sensor.freezer_total_energy', 'previous_input2') }}
            {% endif %}