upgrade causes energy sensor total to be read as day's production

[TheGroundZero]

The problem

Each time I do a Supervisor update (and there've been a few on 2021.9 already), the value of the 'Total' sensor is added as that day's energy consumption.

I have a sensor recording the total energy production of my solar inverter. Prior to 2021.9 I set this up as state_class: measurement with a fake last_reset timestamp set to Unix epoch. After the launch of 2021.9, which introduced the new state_class, I dropped the last_reset and changed the sensor to use state_class: total_increasing. The first day I did this, the Energy Dashboard used the value of the sensor as that day's production. The day after the sensor worked as expected. However, each time I do a Supervisor upgrade, the issue reoccurs.

What is version of Home Assistant Core has the issue?

core-2021.9.4

What was the last working version of Home Assistant Core?

core-2021.8.8

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Energy

Link to integration documentation on our website

https://www.home-assistant.io/docs/energy/

Example YAML snippet

---
# Solis inverter data
#
# https://sequr.be/blog/2021/08/reading-ginlong-solis-inverter-over-serial-and-importing-in-home-assistant-over-mqtt/
#
# [... cut for brevity ...]
- platform: mqtt
  name: "Solis meter - Generated (All time)"
  qos: 1
  unique_id: "solis_com_gat"
  state_topic: "meters/solis_com/gat"
  device_class: energy
  unit_of_measurement: kWh
  state_class: total_increasing
  # 
  # Old config (2021.8.x)
  # state_class: measurement
  # last_reset_topic: 'meters/solis_com/gat'
  # last_reset_value_template: '1970-01-01T00:00:00+00:00'

Anything in the logs that might be useful for us?

No response

Additional information

[mc74909]

I think I have the same issue. But is is not only appearing at the upgrade scenario described here. I have 0-2 times per day that my solar panel production takes a jump.

I have been able to pinpoint is to state updates with a NULL value.

Hope this helps in fixing the issue.

[TheGroundZero]

I have been able to pinpoint is to state updates with a NULL value.

Looking at my sensor, I see an unexplainable dip as well...

Will try to use a filter to filter these out

sensor:
  - platform: filter
    name: "Solis meter - Generated (All time) - clean"
    entity_id: sensor.solis_meter_generated_all_time
    filters:
      - filter: outlier
        window_size: 3
        radius: 10

homeassistant:
  customize:
    sensor.solis_meter_generated_all_time_clean:
      unique_id: "solis_com_gat_clean"
      device_class: energy
      unit_of_measurement: kWh
      state_class: total_increasing

[probot-home-assistant[bot]]

Hey there @emontnemery, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

---

mqtt documentation mqtt source _{^{(message by IssueLinks)}}

[mc74909]

Until fixed, I am fixing my db from time to time with the following script: Maybe others may use too.

import sqlite3

con = sqlite3.connect('home-assistant_v2.db')

ids_cur = con.cursor()

ids = ids_cur.execute("select id from statistics_meta")

update = con.cursor()

for id in ids: statistics_cur = con.cursor() statistics = statistics_cur.execute("select * from statistics where state not NULL and metadata_id = " + str(id[0])) first = True for stat in statistics: if first: previous_state = stat[8] previous_sum = stat[9] first = False new_sum = previous_sum + stat[8] - previous_state if (round(new_sum, 2) != round(stat[9], 2)): print("updating sum from " + str(stat[9]) + " to " + str(new_sum)) update.execute("update statistics set sum=" + str(new_sum) + " where id=" + str(stat[0]))

con.commit() con.close()

[edelans]

Hi there,

I encountered the same problem while setting up my energy dashboard, I've explained it on the issue tracker of the integration https://github.com/Aohzan/ecodevices/issues/13#issue-992114237 that I use to get the readings of my utility counters, I copy here my investigation :

My energy usage looks like this in the dashbaord :

The bars are way too high ! Actually, 12,448 kWh you see in the chart was the reading of my "compteur linky" at that time.

This is because, when you have a closer look at the teleinfo 1 total entity, you see there is a 0kWh sent at some point :

with the total_increasing state_class, what happens is that the energy dashboard thinks I have consummed the 12MWh at that moment, making all other bars of the chart ~0 :

I guess the integration should filter out 0 values (or even filter out decreasing values ?) as suggested in the doc :

My workaround for the moment is to create another sanitized sensor with those following lines in my configuration.yaml file. It filters out decreasing values.

  - sensor:
      name: Elec Consumption Wh increasing only
      state: >
        {% if states('sensor.teleinfo_1_total') > states('sensor.elec_consumption_wh_increasing_only') %}
          {{ states('sensor.teleinfo_1_total') | float(states('sensor.elec_consumption_wh_increasing_only' )) }}
        {% else %} 
          {{ states('sensor.elec_consumption_wh_increasing_only') }}
        {% endif %}
      icon: mdi:flash
      unit_of_measurement: "Wh"
      state_class: "total_increasing"
      device_class: "energy"

[Alex-joomla]

As i described here: https://github.com/home-assistant/core/issues/56022 it's NOT related to MQTT as I use Shellys with CoAP. AND my utility_meter did this occasionally before every now and then.

[probot-home-assistant[bot]]

utility_meter documentation utility_meter source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (utility_meter) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[emontnemery]

The analysis from @edelans is 100% correct.

Integrations should never set a sensor's value to 0 when the state is unknown, for example in the constructor. It should instead set the state to None.

@TheGroundZero in your case you get data from MQTT topic meters/solis_com/gat, can you check if you sometimes receive 0 on that topic? @mc74909 Is your sensor also an MQTT sensor or something else? @Alex-joomla For the template and utility_meter sensor, can you share states from the states table which shows the state of the template and utility_meter + the sensors they depend on when the resets happen?

In Home Assistant 2021.10, state_class total is (re)introduced. This is similar to total_increasing but:

• billing cycles are supported, but managed through the last_reset property
• setting last_reset is optional, if it's not set, the meter will never reset

[dgomes]

@Alex-joomla in your template check availability besides defining state.

Casting an UNAVAILABLE state to int/float leads to 0 (this is a very common reported issue)

[emontnemery]

Thanks @dgomes! I wonder if we should fail on casting random strings to float/int instead of returning 0?

[dgomes]

the issue is that random string = None

[TheGroundZero]

@emontnemery: Integrations should never set a sensor's value to 0 when the state is unknown, for example in the constructor. It should instead set the state to None.

@TheGroundZero in your case you get data from MQTT topic meters/solis_com/gat, can you check if you sometimes receive 0 on that topic?

What's the best method to monitor MQTT messages long-term and store their values? It shouldn't send a 0 as the counter always increases and when there's no sun energy the meter goes offline and the modbus request fails causing the script to abort and no messages to be sent over MQTT.

@emontnemery: In Home Assistant 2021.10, state_class total is (re)introduced. This is similar to total_increasing but:

* billing cycles are supported, but managed through the `last_reset` property

* setting `last_reset` is optional, if it's not set, the meter will **never** reset

How much does this differ from the current total_increasing without last_reset?

Additional question: I have a total_today sensor as well, would it be better to use this one? I'm afraid timezone or summertime differences may cause the total_today sensor to go back to 0 before the end of the Home Assistant day causing weird issues with sensor states.

[Alex-joomla]

Here are Screenshots of the State Graph:

Indeed, there is a little Gap at the point of the failure.

Here is the moment in the SQL (template Sensor of the added Phases) and of course, there is a unknown State and a Zero:

This is the Moment in Utility_Meter:

[emontnemery]

@Alex-joomla

The template sensor

the problem with the template is that | float returns 0 if the value is illegal. Instead of this:

"{{ (states('sensor.shelly_shem_3_8caab5615ec3_1_total_consumption') | float + states('sensor.shelly_shem_3_8caab5615ec3_2_total_consumption') | float + states('sensor.shelly_shem_3_8caab5615ec3_3_total_consumption') | float) | round(2) }}"

Try to do this:

"{{ float((float(states('sensor.shelly_shem_3_8caab5615ec3_1_total_consumption')) + float(states('sensor.shelly_shem_3_8caab5615ec3_2_total_consumption')) + float(states('sensor.shelly_shem_3_8caab5615ec3_3_total_consumption'))) | round(2) }}"

If the values are not valid numbers, this will now fail instead of returning 0.

The utility_meter sensor

For the utility meter, can you share the states of the sensor which the utility meter depends on?

[emontnemery]

@TheGroundZero

I have a total_today sensor as well, would it be better to use this one? I'm afraid timezone or summertime differences may cause the total_today sensor to go back to 0 before the end of the Home Assistant day causing weird issues with sensor states.

If the sensor is marked as total_increasing, Home Assistant will not care about the time; it will detect a reset when the value drops. If the sensor is marked as total or measurement with a valid last_reset, it's important that the value of last_reset changes at exactly the same time as the state resets to 0. Home Assistant will not care about the value of last_reset only about the fact that last_reset changed.

How much does this (total) differ from the current total_increasing without last_reset?

total_increasing -> Every time the sensor's value drops, that is interpreted as a new billing cycle. This is causing the trouble in this issue because the sensor's values are incorrectly decreasing. total + no last_reset -> A new billing cycle will never be detected, the sensor can freely increase or decrease without triggering a new billing cycle. total + last_reset -> A new billing cycle will detected whenever last_reset changes, the sensor can freely increase or decrease without triggering a new billing cycle.

[emontnemery]

What's the best method to monitor MQTT messages long-term and store their values? It shouldn't send a 0 as the counter always increases and when there's no sun energy the meter goes offline and the modbus request fails causing the script to abort and no messages to be sent over MQTT.

I think the easiest way is simply to do: mosquitto_sub -v -h <host> -t <topic>

[mc74909]

@emontnemery yes this is a MQTT sensors

[TheGroundZero]

I think the easiest way is simply to do: mosquitto_sub -v -h <host> -t <topic>

Running mosquito_sub -h <host> -t meters/solis_com/gat -v | while read line; do echo "$(date) - $line; done | tee config/solis_mqtt_gat.log, which gives me outputs like Fri Sep 10 14:15:10 CEST 2021 - meters/solis_com/gat 7139. Will keep it running for 24h or so and see if it reports any non-integer values.

[emontnemery]

@TheGroundZero A non-numeric should perfectly fine, such values are simply ignored by the statistics. There could be bugs though.

[TheGroundZero]

OK, so I've been logging all MQTT messages for the past 72 hours or so.

Noticed a 0 value as the first readings of e.g. 11/09

These 2 were the only occurences between Fri Sep 10 14:15:10 CEST 2021 and Mon Sep 13 09:51:09 CEST 2021

The filtered version of the sensor did prevent the error from propagating. And lucky I chose a window_size of 3 as I got 2 consecutive 0 values

[emontnemery]

Thanks @TheGroundZero for the update! Then I think we can conclude there's a problem with whatever is writing to the meters/solis_com/gat topic?

@mc74909 can you do the same experiment?

[mc74909]

@TheGroundZero started the logging. I haven't seen the issue in 3 days now. So may take a while.

[Alex-joomla]

@emontnemery

The utility_meter sensor

For the utility meter, can you share the states of the sensor which the utility meter depends on?

This is the Template Sensor

[emontnemery]

@Alex-joomla Do you mean the input to the utility_meter sensor is the template sensor? If that's the case, the problem should be avoided by using the float() template function as suggested earlier.

[TheGroundZero]

Is there a method to remove the incorrect values from the DB? If I want to have a look at the Month/Year stats somewhere in the future, it'd of course be nice if these erroneous values would not mess up the totals.

[mc74909]

https://github.com/mc74909/CorrectStatistics/blob/master/CorrectStatistics.py I made a small python script for correcting the statistics.

[Alex-joomla]

@Alex-joomla Do you mean the input to the utility_meter sensor is the template sensor? If that's the case, the problem should be avoided by using the float() template function as suggested earlier.

It happened again :-(

Sensor (template):

      totalverbrauch_haus_kwh:
        friendly_name: 'Energieverbrauch Gesamthaus total'
        entity_id:
        - sensor.shelly_shem_3_8caab5615ec3_1_total_consumption
        - sensor.shelly_shem_3_8caab5615ec3_2_total_consumption
        - sensor.shelly_shem_3_8caab5615ec3_3_total_consumption
        value_template: "{{ float(float(states('sensor.shelly_shem_3_8caab5615ec3_1_total_consumption')) +
                            float(states('sensor.shelly_shem_3_8caab5615ec3_2_total_consumption')) +
                            float(states('sensor.shelly_shem_3_8caab5615ec3_3_total_consumption')))
                            | round(2) }}"

Utility sensor:

  #Stromproduktion Haus gesamt täglich
  energy_energieverbrauch_haus_total_d:
    source: sensor.totalverbrauch_haus_kwh_geglattet
    cycle: daily

tried even a filter in between:

  - platform: filter
    name: "totalverbrauch_haus_kwh_geglättet"
    entity_id: sensor.totalverbrauch_haus_kwh
    filters:
      - filter: outlier
        radius: 100
        window_size: 4

But what happened today:

SQL Data:

hmmmmm

[Alex-joomla]

*made a firmware upgrade of the shelly 3em

[emontnemery]

When the reset happened yesterday, what were the values of the sensor.shelly_shem_3_8caab5615ec3_*_total_consumption ?

[thecode]

@Alex-joomla from the sensor name I understand you use ShellyForHass, maybe the zero value is generated by the integration?

I am one of the code owners for core Shelly integration and using Shelly 3EM myself and did not experience any issues with the Energy sensors.

[emontnemery]

I'm closing this issue since @TheGroundZero's problem has been narrowed down to the device sending unexpected zero-states. @Alex-joomla Please open an issue for your problem with ShellyForHass on the ShellyForHass issue tracker.

Note: In Home Assistant 2021.10 a new state class total is introduced, https://developers.home-assistant.io/blog/2021/09/20/state_class_total, which doesn't have automatic detection of billing cycles.