Utility_meters load current source sensor value at HA reboot

[gio-dot]

The problem

I use 5 utility meters without issues since about 1 year. I feed these meters with time_throttle filtered data from sonoff pow r2 mqtt tasmota. I use only 1 tariff in all these meters. Recently (i'm not sure when exactly, i think about 2 months) i noticed that at reboot, 2 of them load current source sensor value at HA reboot on all the periods (week, month, year).

Environment

• Raspberry PI3B+
• Home Assistant Core release with the issue: 0.113.2
• Last working Home Assistant Core release (if known): -
• Operating environment (OS/Container/Supervised/Core): Home Assistant OS
• Integration causing this issue:
• Link to integration documentation on our website:

arch	armv7l
chassis	embedded
dev	false
docker	true
docker_version	19.03.8
hassio	true
host_os	HassOS 4.11
installation_type	Home Assistant OS
os_name	Linux
os_version	4.19.127-v7
python_version	3.8.3
supervisor	229
timezone	Europe/Rome
version	0.113.2
virtualenv	false

Problem-relevant configuration.yaml

sensor:
 - platform: filter  
     name: "Main-1P-kWh-slow"  
     entity_id: sensor.main_1p_kwh 
     filters:
       - filter: time_throttle 
         window_size: 00:10  

utility_meter:
 main_1p_kwh_week:  
    source: sensor.main_1p_kwh_slow
    cycle: weekly
    tariffs:
      - peak    
  main_1p_kwh_month:  
    source: sensor.main_1p_kwh_slow
    cycle: monthly
    tariffs:
      - peak
  main_1p_kwh_year:  
    source: sensor.main_1p_kwh_slow
    cycle: yearly
    tariffs:
      - peak    

Traceback/Error logs

Additional information

Below there is a screenshot of 1 utility meter after an HA reboot. Before i reset all that meters using service utility_meter.reset. All (week/month/year was at 0 before reboot). I tried even utility_meter.calibrate: 0 before reboot -> 2623.x kWh after. I use other utility_meters with similar source (time throttle filtered values) that don'thave any issue. I can't find any difference in my configuration between working and bad working meters...

[probot-home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as its been labeled with an integration (utility_meter) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

[dgomes]

Can you share history graphs of sensor.main_1p_kwh_slow and sensor.main_1p_kwh_week?

[gio-dot]

Can you share history graphs of sensor.main_1p_kwh_slow and sensor.main_1p_kwh_week?

Ok, it seems that time throttle filter is the cause of the issue; below there is the filter output at reboot (tried 4 reboot). I think it would be useful to make utility meters startup in somehow delayed respect to all other sensors to avoid this issues. Strange thing is that i have other time throttle sensors that feed other utility meters without issues. Maybe the issue is due to a combination of original sensor type + time throttle filter???

[probot-home-assistant[bot]]

filter documentation filter source _{^{(message by IssueLinks)}}

[dgomes]

OK, so now I need the history graph of sensor.main_1p_kwh

[gio-dot]

[gio-dot]

Source sensor don't produce any spyke, time throttle filtered sensor instead seems to produce spykes at HA reboot. Entity of these spikes seems to be also bigger if the source sensor (the one not filtered) is excluded from recorder. In this case the spike size is given by the difference from current filtered sensor value and last value of the not filtered sensor in the recorder.

[gio-dot]

In general I also noticed that all filtered sensors (ie average etc.) starts at a wrong values at HA reboot. Below an example of a filtered temperature sensor at reboot. I think that, if possible, a mechanism that prevent wrong values of filtered sensors at reboot should be implemented. Reboot spikes:

[dgomes]

filters at reboot will only work properly if the source sensor has been saved (recorder component must be active), else they enter an initial warmup phase that will always lead to inevitable errors while the filter buffers are filled with proper values.

In your graphs, at what time was the reboot ?

[prilly-dev]

I have the same issue, more or less.

I am using fibaro dimmers that is providing entity energy, this is a incrementing meter of consumed energy in kWh, it is persistant and is always counting up without possibility to soft reset.

when i am using energy entity with utility_meter everything works fine until you restart ha. Utility meter is saving last value of the meter in core.restore_state, when HA restarts the energy entity value is added to the saved state value resulting in calculation error. for each restart it add the value of the energy entity.

Something is clearly off here.

What i have noticed is that the value stored in core.restore_state is the same as the the energy entity value. When restart this is added to the energy entity and you see a doubling in total value

[simonbaudart]

Hello !

I have the same issue, but with template sensor.

Here is my template sensor :

and the utility_meter:

The first utility meter, daily_energy_wp, is working well on each reboot. The second, daily_energy, is adding last value at each reboot.

The difference is that the custom_sensor_energy_total seems to start at 0 when HA reboot, for a few secondes (visible in history)

Is it possible to ignore 0 value from source ? Or is this an issue with my configuration ? I don't have a global sensor for energy usage, so I have to sum all my sensor and cannot find another way to do it ...

Regards,

Simon

[Kirorus]

i have same the issue. I sum all my shelly consumption data with template

sensor:
  - platform: template
    sensors:
      total_electricity_consumption:
          friendly_name: Общее Потребление Электричества
          entity_id:
            - sensor.time
          unit_of_measurement: kWh
          value_template: >
            {% set ns = namespace(states=[]) %}
            {% for s in states.sensor %}
              {% if s.object_id.startswith('shelly_sh') and s.object_id.endswith('_total_consumption') %}
                {% set ns.states = ns.states + [ s.state | float ] %}
              {% endif %}
            {% endfor %}
            {{ ns.states | sum | round(2) }}

and utility meter:

utility_meter:
  total_electricity_hourly_consumprion:
    source: sensor.total_electricity_consumption
    cycle: hourly
    tariffs:
      - offpeak
      - halfpeak
      - peak
  total_electricity_daily_consumprion:
    source: sensor.total_electricity_consumption
    cycle: daily
    tariffs:
      - offpeak
      - halfpeak
      - peak
  total_electricity_weekly_consumprion:
    source: sensor.total_electricity_consumption
    cycle: weekly
    tariffs:
      - offpeak
      - halfpeak
      - peak
  total_electricity_monthly_consumprion:
    source: sensor.total_electricity_consumption
    cycle: monthly
    tariffs:
      - offpeak
      - halfpeak
      - peak
  total_electricity_quarterly_consumprion:
    source: sensor.total_electricity_consumption
    cycle: quarterly
    tariffs:
      - offpeak
      - halfpeak
      - peak
  total_electricity_yearly_consumprion:
    source: sensor.total_electricity_consumption
    cycle: yearly
    tariffs:
      - offpeak
      - halfpeak
      - peak

[prilly-dev]

Its exactly the same scrip i use for adding up multiple energy entitis.

So the problem seems to be the utility meter starts calculation before the sensor template has finnished adding stuff togheter, resulting i 0 values.

It should be possible to delay the utility calculation until template sensor is finnished.

[simonbaudart]

I got a workaround, but this is a verrrrrrrrrrrrrrrrrrry long workaround ... I try the filter sensor, but it doesn't work.

So, I configured a mqtt sensor, and an automation. The automation publish a mqtt message on the sensor topic when the template sensor is above 1 for at least 30 seconds. The state of the mqtt sensor is "Unknown" during the startup, and the utility meters ignore this.

The template sensor :

The mqtt sensor :

The utility meter with the mqtt sensor as source :

And, finally, the automation:

/brainfuck

[prilly-dev]

Yeah that is also a way to do it, a long way.

I am looking at availability_template, it might solve this 0 at startup issue, what you think?

[simonbaudart]

does it work?

I don't know how to use it in the current case ... for comparison, I understand, but will it work for template ? I think is_state() returns a boolean and not a numeric ...

Yeah that is also a way to do it, a long way.

I am looking at availability_template, it might solve this 0 at startup issue, what you think?

Maybe, with the is_state and returning true after checking all the entities or returning false at the end. You're right, I think this is the way to say that the template sensor is in an unknown state ...

[Kirorus]

yes, you are right... how i can see template sensor is not unknown at start... it grow from 0 to real value.

[simonbaudart]

GOT IT WORKING !!! Thanks @prilly-dev and @Kirorus !

      custom_sensor_energy_total:
        friendly_name: "Consommation totale"
        unit_of_measurement: 'kWh'
        value_template: "{{ 
                            states('sensor.fibaro_system_fgwpe_f_wall_plug_media_center_energy') | float | round(2)
                          + states('sensor.fibaro_system_fgwpe_f_wall_plug_salon_energy') | float | round(2)
                          + states('sensor.fibaro_system_fgwpe_f_wall_plug_frigo_energy') | float | round(2)
                          }}"
        availability_template: >
          {% if is_state("sensor.fibaro_system_fgwpe_f_wall_plug_media_center_energy", "unavailable") %}
            false
          {% elif is_state("sensor.fibaro_system_fgwpe_f_wall_plug_salon_energy", "unavailable") %}
            false
          {% elif is_state("sensor.fibaro_system_fgwpe_f_wall_plug_frigo_energy", "unavailable") %}
            false
          {% else %}
            true
          {%- endif %}

This way, the custom sensor in unavailable until each sensor is available.

EDIT: I'll clean up my config and maybe use a loop like @Kirorus posted.

[Kirorus]

EDIT: I'll clean up my config and maybe use a loop like @Kirorus posted.

yes it is good way, but how to adopt your "availability_template"

[prilly-dev]

Yeah great.

Kirorus uses a for loop to collect entitys to be included, they are not defined one by one. This sensor is also using sensor.time to trigger first calculation.

What happens with the value of template sensor until first calculation, unavailable?

And what entity to test against in the avaiøability template? Use same for loop?

[prilly-dev]

Kirorus have a look here: https://community.home-assistant.io/t/utility-meter-wrong-calculation/227293/7

[Kirorus]

@prilly-dev , @simonbaudart thank!

[simonbaudart]

Easy to maintain : no yaml edition when you add a sensor Drawback : you have to wait one minute maximum before the sensor is up, it's greyed out. But, works with utility meter !

I'll keep this version. One more thanks @prilly-dev , @Kirorus !

      custom_sensor_energy_total:
        friendly_name: "Consommation totale"
        entity_id: sensor.time
        unit_of_measurement: 'kWh'
        value_template: >
          {% set ns = namespace(states=[]) %}
          {% for s in states.sensor %}
            {% if s.entity_id.startswith('sensor.') and s.entity_id.endswith('_energy') %}
              {% set ns.states = ns.states + [ s.state | float ] %}
            {% endif %}
          {% endfor %}
          {{ ns.states | sum | round(2) }}
        availability_template: >
          {% set ns = namespace(offline=0) %}
          {% for s in states.sensor %}
            {% if s.entity_id.startswith('sensor.') and s.entity_id.endswith('_energy') and s.state in ['unavailable', 'unknown' ] %}
              {% set ns.offline = ns.offline + 1 %}
            {% endif %}
          {% endfor %}
          {{ ns.offline == 0 }}

[prilly-dev]

Conclusion: utility_meter works as it should. Use availability_template with template sensor to avoid "0 state at startup" by setting template sensor to unavailable until calculasion is done.

After ha version 115 entity_id: sensor.time can be removed.

[dgomes]

Guys, great job tracing the issue 👍

Can we close down the issue now ?

[Kirorus]

@dgomes I think yes. I test this at 114 and 115 beta. Everything work!

this is my template sensor:

  - platform: template
    sensors:
      total_electricity_consumption:
          friendly_name: Общее Потребление Электричества
# Uncomment this for 114 and lower
#          entity_id:
#            - sensor.time
          unit_of_measurement: kWh
          value_template: >
            {% set ns = namespace(states=[]) %}
            {% for s in states.sensor %}
              {% if s.object_id.startswith('shelly_sh') and s.object_id.endswith('_total_consumption') %}
                {% set ns.states = ns.states + [ s.state | float ] %}
              {% endif %}
            {% endfor %}
            {{ ns.states | sum | round(2) }}
          availability_template: >
            {% set ns = namespace(offline=0) %}
            {% for s in states.sensor %}
              {% if s.object_id.startswith('shelly_sh') and s.object_id.endswith('_total_consumption') and s.state in ['unavailable', 'unknown' ] %}
                {% set ns.offline = ns.offline + 1 %}
              {% endif %}
            {% endfor %}
            {{ ns.offline == 0 }}

[prilly-dev]

Yes i think it can be closed.

[simonbaudart]

I agree, for me the issue is solved. Maybe, add the information in the utility meter documentation as a note :)

Thanks all for the team work ! I'm new to home assistant and really like the community !

See you !

[iz3man]

Excuse me, but shouldn't that be included in the utility meter code internally? If I look at the code that needs to be written to avoid those issues I'm overwhelmed. This is way too much code for a regular "user" to make this work. Or did I misunderstood something?

This is what the meter looks like NOW

And this is what it looks like after a reboot:

And after a new value is received it's messed up: