Big spike in utility meter value while energy value used as source stays constant

ASL07 commented 4 years ago

The problem

Hi,

I am using utility meter for recording monthly and daily energy consumption. I have several meters for several apartments as well. The value comes from a pzem004-t. It usually works fine, but sometimes, I get a huge spike in the utility meter (please the pzem-004t doesn’t show any spike, so I am quite sure the problem lies in utility meter).

Please see a screenshot below, there is one big spike in utility meter at 16:17 (from 250kwh to 409 kwh), and another one at 20:12(from 409kwh to 800kwh), while the energy value which is used as a source for the utility meter stays constant. The graph only shows one meter for clarity, but the same happened to all the other meters

Environment

Home assistant version :

Home Assistant Core release with the issue: 0.99.2
Last working Home Assistant Core release (if known):
Operating environment (Home Assistant/Supervised/Docker/venv): Raspbian(venv)
Integration causing this issue: Utility meter
Link to integration documentation on our website: https://www.home-assistant.io/integrations/utility_meter/

Problem-relevant `configuration.yaml`

utility_meter:
  apartment_1_montly_energy_kwh:
    source: sensor.apartment_1_energy_kwh
    cycle: monthly
  apartment_1_daily_energy_kwh:
    source: sensor.apartment_1_energy_kwh
    cycle: daily
  apartment_2_montly_energy_kwh:
    source: sensor.apartment_2_energy_kwh
    cycle: monthly
  apartment_2_daily_energy_kwh:
    source: sensor.apartment_2_energy_kwh
    cycle: daily
  apartment_3_montly_energy_kwh:
    source: sensor.apartment_3_energy_kwh
    cycle: monthly
  apartment_3_daily_energy_kwh:
    source: sensor.apartment_3_energy_kwh
    cycle: daily
  apartment_4_montly_energy_kwh:
    source: sensor.apartment_4_energy_kwh
    cycle: monthly
  apartment_4_daily_energy_kwh:
    source: sensor.apartment_4_energy_kwh
    cycle: daily
  apartment_5_montly_energy_kwh:
    source: sensor.apartment_5_energy_kwh
    cycle: monthly
  apartment_5_daily_energy_kwh:
    source: sensor.apartment_5_energy_kwh
    cycle: daily
  apartment_6_montly_energy_kwh:
    source: sensor.apartment_6_energy_kwh
    cycle: monthly
  apartment_6_daily_energy_kwh:
    source: sensor.apartment_6_energy_kwh
    cycle: daily
  apartment_7_montly_energy_kwh:
    source: sensor.apartment_7_energy_kwh
    cycle: monthly
  apartment_7_daily_energy_kwh:
    source: sensor.apartment_7_energy_kwh
    cycle: daily
  apartment_8_montly_energy_kwh:
    source: sensor.apartment_8_energy_kwh
    cycle: monthly
  apartment_8_daily_energy_kwh:
    source: sensor.apartment_8_energy_kwh
    cycle: daily

Traceback/Error logs

No errors are present in ha logs

Additional information

Thanks for your help

probot-home-assistant[bot] commented 4 years ago

Hey there @dgomes, mind taking a look at this issue as its been labeled with a integration (utility_meter) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

dgomes commented 4 years ago

Can you please show the source graphs?

ASL07 commented 4 years ago

Hi @dgomes . I provide more details about the issue: Upon closer inspection, I have found that the spikes were triggered by two momentaneous drops to 0 in the source value. See the drops here, the timing matches the raise in utility meter value :

Daily utility meter graph:

Monthly utility meter graph:

Configuration for source sensor:

  - platform: mqtt
    name: "Apartment 1 energy"
    state_topic: "panel/sensor/apartment_1_energy/state"
    availability_topic: "panel/status"
    unit_of_measurement: "Wh"

Template sensor to convert value to kwh:

  - platform: template
    sensors:
      apartment_1_energy_kwh:
        entity_id: sensor.apartment_1_energy
        value_template: "{{ states('sensor.apartment_1_energy') | float / 1000 }}"
        friendly_name: "Apartment 1 energy kWh"
        unit_of_measurement: kWh

Utility meter config:

utility_meter:
  apartment_1_montly_energy_kwh:
    source: sensor.apartment_1_energy_kwh
    cycle: monthly
  apartment_1_daily_energy_kwh:
    source: sensor.apartment_1_energy_kwh
    cycle: daily

dgomes commented 4 years ago

So there you have it 😄 it's not utility meter fault!

I advise you to add a filter sensor (https://www.home-assistant.io/integrations/filter/) to avoid those problems in the future.

ASL07 commented 4 years ago

I know the sensor failed, I will try to use a filter. However, I belive there is something wrong with the utility meter as well, a sudden 0 value shouldn't affect the measurement of the meter.

In my understanding, the utility meter value is calculated like this: utility_meter_value = (previous_utility_meter_value) > (sensor value - utility_meter_cycle_initial_value) ? previous_utility_meter_value : (sensor value - utility_meter_cycle_initial_value)

Where utility_meter_cycle_initial_value = Last sensor value of the previous cycle

But in my scenario, it doesn't work like that, let me clarify with an example of what happens in my installation: Start of day one Sensor value: 0kWh Utility meter cycle initial value: 0kWh Utility meter value: 0kWh At the end of the day 1: Sensor value: 10kWh Utility meter value: 10kWh

Start of Day 2: Sensor value: 10kWh Utility meter cycle initial value: 10kWh Utility meter value: 0kWh (10kWh -10kWh) After some time: Sensor value: 15kWh Utility meter value: 5kWh (15kWh -10kWh) Next sensor update: Sensor value: 0 (We get a faulty 0 reading from the sensor) Utility meter value: 5kWh (previous value) Next sensor update: Sensor value: 16kWh Expected utility meter value: 16kWh (16kWh - 10kWh) Actual utility meter value: 31kWh (16kWh + 15kWh)

Could you please correct me if I'm wrong? Or else explain how the utility meter value is calculated?

Thanks for your time

dgomes commented 4 years ago

A sudden zero does affect utility_meter:

https://github.com/home-assistant/core/blob/dev/homeassistant/components/utility_meter/sensor.py#L153

we work with small increments based on the values of the source sensor (this makes it possible to reset counters in the source and don't lose too much data)

So in your case: old_state is 0kWh new state is back to 110kWh, that creates an increase in the meter of 110kWh (the jump you see)

ASL07 commented 4 years ago

@dgomes After looking at the filters documentation as you suggested. I cannot see any particular filter which can be applied to my case (removing "0" values). What kind of filter would you suggest for this?

dgomes commented 4 years ago

You can use the "outlier" or "range" filter

If your pretty sure the 0 value is not "possible" you can setup "range", using 0 as the lower bound.

Else try to figure out was it the largest spike, and use it to configure "outlier"

ASL07 commented 4 years ago

Thank you

From range filter docs:

All values greater then the upper bound are replaced by the upper bound and all values lower than the lower bound are replaced by the lower bound. Per default there are neither upper nor lower bound.

In this case, if I use range with 0 as lower bound, any value below 0 will be replaced by 0, I don't want 0 to be replaced by the lower bound, I want it to be replaced by the previous value, or discarded.

Outlier filter just checks if the value differs from a median of the previous values. This should work for example, for temperature sensors, because the temperature changes gradually. But, since power consumption can vary greatly between consecutive measurements, I don't think this will apply to my case

My workaround was setting a filter in the sensor firmware, so it doensn't send 0 values (I am using esphome). But it would be good to have something in homeassistant to filter specific values as well

dgomes commented 4 years ago

@ASL07 you are right

Maybe you can suggest a new filter or open a new PR yourself :) ?

KevinD-87 commented 3 years ago

@ASL07 Did you find a solution for your problem? I'm having the same issue with my Shelly energy meter sometimes sending a 0 value, which messes up my daily and monthly consumption stats.

ASL07 commented 3 years ago

I think I fixed it, or at least mitigated it, by adding a template sensor which removed the 0 values But I am not 100% sure how, this was along time ago I don't have access to that particular home assistant installation anymore

Please have a look at my posts in the home assistant community forum: https://community.home-assistant.io/t/i-need-to-remove-zero-from-the-sensor/152429/9 https://community.home-assistant.io/t/template-sensor-error-state-attr-missing-1-required-positional-argument-name/191045/3

I hope it helps

KevinD-87 commented 3 years ago

I think I fixed it, or at least mitigated it, by adding a template sensor which removed the 0 values But I am not 100% sure how, this was along time ago I don't have access to that particular home assistant installation anymore

Please have a look at my posts in the home assistant community forum: https://community.home-assistant.io/t/i-need-to-remove-zero-from-the-sensor/152429/9 https://community.home-assistant.io/t/template-sensor-error-state-attr-missing-1-required-positional-argument-name/191045/3

I hope it helps

Thank you for the quick reply. The first link you posted seems to be what I needed. From a small manual test where I rebooted my HA service, I saw the original sensor drop to 0, but the filtered sensor didn't. Thanks!

ChirpyTurnip commented 3 years ago

Thanks! I have the same problem on my end.....though I'm not sure where the zero value comes from....it's not coming from the sensors as the trace there is clear with no drops.....it looks to me like it comes from my template_sensor which is aggregating the data (I need to do some adding an subtracting to break my sensor sensor data into a service-based output). It drops to zero several times a week and then comes straight back....but not before wreaking havoc on the utility meter. I don't know why it does this as there doesn't seem to be any clear pattern.... :-(

I'll put your solution on top of the aggregation function to see if I can suppress this, but I do think that consideration should be given to applying a filter to the utility meter to prevent these spikes. Power consumption, except in extreme industrial applications, is a gradual increase over time....any significant step change where there is a jump of say, 90kWh, from one reading to the next simply has to be an error....and therefore should be probably be suppressed as an errant input. The utility meter is very cool but I'm increasingly getting the sense that if I can't get the data inputs to be 100% reliable I won't be able to rely on it....

KevinD-87 commented 3 years ago

Thanks! I have the same problem on my end.....though I'm not sure where the zero value comes from....it's not coming from the sensors as the trace there is clear with no drops.....it looks to me like it comes from my template_sensor which is aggregating the data (I need to do some adding an subtracting to break my sensor sensor data into a service-based output). It drops to zero several times a week and then comes straight back....but not before wreaking havoc on the utility meter. I don't know why it does this as there doesn't seem to be any clear pattern.... :-(

I'll put your solution on top of the aggregation function to see if I can suppress this, but I do think that consideration should be given to applying a filter to the utility meter to prevent these spikes. Power consumption, except in extreme industrial applications, is a gradual increase over time....any significant step change where there is a jump of say, 90kWh, from one reading to the next simply has to be an error....and therefore should be probably be suppressed as an errant input. The utility meter is very cool but I'm increasingly getting the sense that if I can't get the data inputs to be 100% reliable I won't be able to rely on it....

I understand why they wouldn't want to add such a filter though. Maybe some sensors reset to 0 from time to time and as such, a drop is valid. Maybe you have a sensor that only reports data a few times a year and as such a big jump is normal. Libraries and plugins should often be as 'dumb' as possible, because they don't know who or what is providing input for them. Maybe an optional, configurable filter would be nice. But it's possible to use other solutions for this like the solution from ASL07 above, so I guess it's not a huge priority to add this to utility meter. If I actually knew Python I might have tried adding something and issue a pull request. But so far, the filter has worked perfectly; my consumption spikes from the sensor dropping to 0 have disappeared.

home-assistant / core