Closed ASL07 closed 4 years ago
Hey there @dgomes, mind taking a look at this issue as its been labeled with a integration (utility_meter
) you are listed as a codeowner for? Thanks!
(message by CodeOwnersMention)
Can you please show the source graphs?
Hi @dgomes . I provide more details about the issue:
Upon closer inspection, I have found that the spikes were triggered by two momentaneous drops to 0 in the source value.
See the drops here, the timing matches the raise in utility meter value :
Daily utility meter graph:
Monthly utility meter graph:
Configuration for source sensor:
- platform: mqtt
name: "Apartment 1 energy"
state_topic: "panel/sensor/apartment_1_energy/state"
availability_topic: "panel/status"
unit_of_measurement: "Wh"
Template sensor to convert value to kwh:
- platform: template
sensors:
apartment_1_energy_kwh:
entity_id: sensor.apartment_1_energy
value_template: "{{ states('sensor.apartment_1_energy') | float / 1000 }}"
friendly_name: "Apartment 1 energy kWh"
unit_of_measurement: kWh
Utility meter config:
utility_meter:
apartment_1_montly_energy_kwh:
source: sensor.apartment_1_energy_kwh
cycle: monthly
apartment_1_daily_energy_kwh:
source: sensor.apartment_1_energy_kwh
cycle: daily
So there you have it 😄 it's not utility meter fault!
I advise you to add a filter sensor (https://www.home-assistant.io/integrations/filter/) to avoid those problems in the future.
I know the sensor failed, I will try to use a filter. However, I belive there is something wrong with the utility meter as well, a sudden 0 value shouldn't affect the measurement of the meter.
In my understanding, the utility meter value is calculated like this:
utility_meter_value = (previous_utility_meter_value) > (sensor value - utility_meter_cycle_initial_value) ? previous_utility_meter_value : (sensor value - utility_meter_cycle_initial_value)
Where utility_meter_cycle_initial_value = Last sensor value of the previous cycle
But in my scenario, it doesn't work like that, let me clarify with an example of what happens in my installation: Start of day one Sensor value: 0kWh Utility meter cycle initial value: 0kWh Utility meter value: 0kWh At the end of the day 1: Sensor value: 10kWh Utility meter value: 10kWh
Start of Day 2: Sensor value: 10kWh Utility meter cycle initial value: 10kWh Utility meter value: 0kWh (10kWh -10kWh) After some time: Sensor value: 15kWh Utility meter value: 5kWh (15kWh -10kWh) Next sensor update: Sensor value: 0 (We get a faulty 0 reading from the sensor) Utility meter value: 5kWh (previous value) Next sensor update: Sensor value: 16kWh Expected utility meter value: 16kWh (16kWh - 10kWh) Actual utility meter value: 31kWh (16kWh + 15kWh)
Could you please correct me if I'm wrong? Or else explain how the utility meter value is calculated?
Thanks for your time
A sudden zero does affect utility_meter:
we work with small increments based on the values of the source sensor (this makes it possible to reset counters in the source and don't lose too much data)
So in your case: old_state is 0kWh new state is back to 110kWh, that creates an increase in the meter of 110kWh (the jump you see)
@dgomes After looking at the filters documentation as you suggested. I cannot see any particular filter which can be applied to my case (removing "0" values). What kind of filter would you suggest for this?
You can use the "outlier" or "range" filter
If your pretty sure the 0 value is not "possible" you can setup "range", using 0 as the lower bound.
Else try to figure out was it the largest spike, and use it to configure "outlier"
Thank you
From range filter docs:
All values greater then the upper bound are replaced by the upper bound and all values lower than the lower bound are replaced by the lower bound. Per default there are neither upper nor lower bound.
In this case, if I use range with 0 as lower bound, any value below 0 will be replaced by 0, I don't want 0 to be replaced by the lower bound, I want it to be replaced by the previous value, or discarded.
Outlier filter just checks if the value differs from a median of the previous values. This should work for example, for temperature sensors, because the temperature changes gradually. But, since power consumption can vary greatly between consecutive measurements, I don't think this will apply to my case
My workaround was setting a filter in the sensor firmware, so it doensn't send 0 values (I am using esphome). But it would be good to have something in homeassistant to filter specific values as well
@ASL07 you are right
Maybe you can suggest a new filter or open a new PR yourself :) ?
@ASL07 Did you find a solution for your problem? I'm having the same issue with my Shelly energy meter sometimes sending a 0 value, which messes up my daily and monthly consumption stats.
I think I fixed it, or at least mitigated it, by adding a template sensor which removed the 0 values But I am not 100% sure how, this was along time ago I don't have access to that particular home assistant installation anymore
Please have a look at my posts in the home assistant community forum: https://community.home-assistant.io/t/i-need-to-remove-zero-from-the-sensor/152429/9 https://community.home-assistant.io/t/template-sensor-error-state-attr-missing-1-required-positional-argument-name/191045/3
I hope it helps
I think I fixed it, or at least mitigated it, by adding a template sensor which removed the 0 values But I am not 100% sure how, this was along time ago I don't have access to that particular home assistant installation anymore
Please have a look at my posts in the home assistant community forum: https://community.home-assistant.io/t/i-need-to-remove-zero-from-the-sensor/152429/9 https://community.home-assistant.io/t/template-sensor-error-state-attr-missing-1-required-positional-argument-name/191045/3
I hope it helps
Thank you for the quick reply. The first link you posted seems to be what I needed. From a small manual test where I rebooted my HA service, I saw the original sensor drop to 0, but the filtered sensor didn't. Thanks!
Thanks! I have the same problem on my end.....though I'm not sure where the zero value comes from....it's not coming from the sensors as the trace there is clear with no drops.....it looks to me like it comes from my template_sensor which is aggregating the data (I need to do some adding an subtracting to break my sensor sensor data into a service-based output). It drops to zero several times a week and then comes straight back....but not before wreaking havoc on the utility meter. I don't know why it does this as there doesn't seem to be any clear pattern.... :-(
I'll put your solution on top of the aggregation function to see if I can suppress this, but I do think that consideration should be given to applying a filter to the utility meter to prevent these spikes. Power consumption, except in extreme industrial applications, is a gradual increase over time....any significant step change where there is a jump of say, 90kWh, from one reading to the next simply has to be an error....and therefore should be probably be suppressed as an errant input. The utility meter is very cool but I'm increasingly getting the sense that if I can't get the data inputs to be 100% reliable I won't be able to rely on it....
Thanks! I have the same problem on my end.....though I'm not sure where the zero value comes from....it's not coming from the sensors as the trace there is clear with no drops.....it looks to me like it comes from my template_sensor which is aggregating the data (I need to do some adding an subtracting to break my sensor sensor data into a service-based output). It drops to zero several times a week and then comes straight back....but not before wreaking havoc on the utility meter. I don't know why it does this as there doesn't seem to be any clear pattern.... :-(
I'll put your solution on top of the aggregation function to see if I can suppress this, but I do think that consideration should be given to applying a filter to the utility meter to prevent these spikes. Power consumption, except in extreme industrial applications, is a gradual increase over time....any significant step change where there is a jump of say, 90kWh, from one reading to the next simply has to be an error....and therefore should be probably be suppressed as an errant input. The utility meter is very cool but I'm increasingly getting the sense that if I can't get the data inputs to be 100% reliable I won't be able to rely on it....
I understand why they wouldn't want to add such a filter though. Maybe some sensors reset to 0 from time to time and as such, a drop is valid. Maybe you have a sensor that only reports data a few times a year and as such a big jump is normal. Libraries and plugins should often be as 'dumb' as possible, because they don't know who or what is providing input for them. Maybe an optional, configurable filter would be nice. But it's possible to use other solutions for this like the solution from ASL07 above, so I guess it's not a huge priority to add this to utility meter. If I actually knew Python I might have tried adding something and issue a pull request. But so far, the filter has worked perfectly; my consumption spikes from the sensor dropping to 0 have disappeared.
The problem
Hi,
I am using utility meter for recording monthly and daily energy consumption. I have several meters for several apartments as well. The value comes from a pzem004-t. It usually works fine, but sometimes, I get a huge spike in the utility meter (please the pzem-004t doesn’t show any spike, so I am quite sure the problem lies in utility meter).
Please see a screenshot below, there is one big spike in utility meter at 16:17 (from 250kwh to 409 kwh), and another one at 20:12(from 409kwh to 800kwh), while the energy value which is used as a source for the utility meter stays constant. The graph only shows one meter for clarity, but the same happened to all the other meters
Environment
Home assistant version :
Problem-relevant
configuration.yaml
Traceback/Error logs
Additional information
Thanks for your help