Utility meter doesn't take value that changed during reboot

Matssa56 commented 10 months ago

The problem

Hi,

I have an ESP that acts like an energy meter of the whole house (directly on the mains on the house between me and my provider). It sends data through MQTT to Home Assistant every 30mins. This data is then used in the Energy Dashboard.

I reboot my Home Assistant VM every night in order to start fresh. During the reboot, it is possible to have a value change on my mains and read by the ESP. However since the MQTT server is down, no message is sent. When the server boots up and when the ESP sends the new value, the Energy Dashboard manages to make the difference between the old value that it had in the database and the new value. Therefore, I don't lose any data.

In my country, we can have contracts where during a certain period of the day, the cost is either higher or lower. In order to see if it cost effective, I created a utility meter through the front-end using the MQTT sensor as an input. I also have an automation that "selects" the type of the utility meter (either high or low) at 2 specific hours. I then use these two sensors in the Energy Dashboard so that I can see the total house consumption, the low consumption and the high consumption (with the costs and all).

This works fine, except during the reboot during the night. Indeed, the utility meter doesn't seem to calculate the change that there was before and after the reboot. So I get a difference between the complete consumption and low or high consumption (which should be the same).

Here is an image showing the issue. The reboot occurs between 6 and 7 am, the dark blue is the total sensor and the light blue the utility meter sensor.

What version of Home Assistant Core has the issue?

core-2023.10.3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

utility_meter

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

Matssa56 commented 10 months ago

Hi, quick update, the sensor that I use for the utility meter doesn't reset when there is the reboot, so no reset to 0 stuff that could interfere.

However there are missing values at the reboot (the history graph has holes in it), would that be the isue? I set the retain to 1 so there shouldn't be any issues, otherwise the energy graph would have the same issue:

home-assistant[bot] commented 10 months ago

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (utility_meter) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `utility_meter` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign utility_meter` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

utility_meter documentation utility_meter source _{^{(message by IssueLinks)}}

dgomes commented 10 months ago

The issue is indeed that period of time the source is off.

You can use this configuration option to address that use-case: https://www.home-assistant.io/integrations/utility_meter/#periodically_resetting

Matssa56 commented 10 months ago

Arg thanks and sorry for this non issue... Should read more carefully... Cheers!

Matssa56 commented 10 months ago

@dgomes FYI the setting didn't fix my issue, I'm still missing values in my energy dashboard between 6 and 7 am.

Matssa56 commented 10 months ago

This setting was disabled at first, after your comment I activated the setting and it didn't change anything. I'm deactivating again to see since the sensor value doesn't go down to 0.

dgomes commented 10 months ago

What about "availability" of the source sensor ?

Matssa56 commented 10 months ago

The source sensor is a MQTT sensor which has the retain flag. However check the image on the second post, we can see that the history graph is empty when the reboot occurs. Not sure how I can change the availability of this sensor though.

dgomes commented 10 months ago

There simply should not be any gap in the source sensor...

But the utility_meter is capable of checking if the source sensor is available or not, and act accordingly. And the periodically_resetting: false configuration option should catch up the values lost during the reboot...

Matssa56 commented 10 months ago

For me these gaps are linked to :

the time that the system takes to boot up proxmox + the VM
the time to turn on the EMQX broker
the time to get the data from MySQL database and refresh everything

There is also the time the device takes to send the new values, I set the esp to send every 10min. However the retain flag should help.

When checking the graph, I see about 20min where the sensor doesn't have any value, not even 0 (null). Would it be the null value that is bugging the system? On the energy dashboard, I don't have any issues getting the correct values.

Matssa56 commented 10 months ago

How can I check the availability parameter of the sensor/device giving the data? Maybe it's always available for the utility meter.

dgomes commented 10 months ago

Null value does create issues! If you can set the availability of the sensor, that would certainly fix the issue:

https://www.home-assistant.io/integrations/sensor.mqtt/#availability

Matssa56 commented 10 months ago

Null value does create issues! If you can set the availability of the sensor, that would certainly fix the issue:

https://www.home-assistant.io/integrations/sensor.mqtt/#availability

Thanks however the sensor is made through the MQTT broker, not via yaml config. Not sure how I can change the availability, maybe through customize.yaml?

Matssa56 commented 10 months ago

Just thought of that, my esp goes dark every 10min in deep sleep (disconnected from wifi) and I don't have any missing values. Technically, it's the same reaction when I reboot the server... So setting another availability won't really change much imo. Furthermore I would have issues on the energy dashboard as well imo.

dgomes commented 10 months ago

I get it... you reboot the MQTT broker... not the sensor... You should skip that one 😓

Matssa56 commented 10 months ago

I get it... you reboot the MQTT broker... not the sensor... You should skip that one 😓

Mmmh gonna be hard since HA has the mqtt as a supervised addon ^^' I tried to add another utility meter through yaml directly, gonna see if it's better. Gonna check the values as well, maybe it's a energy dashboard issue, not a utility meter issue 🤔

dgomes commented 10 months ago

through yaml or UI it's the same.

Wonder what state is your MQTT sensor during boot

Matssa56 commented 10 months ago

I'll try to check tomorrow and force some HA reboots and see how it goes. I did check the values and here are the results : Source sensor delta Utility meter delta 06:00 12594,16 623,76
06:25 12594,26 0,1 623,76 0 06:55 12594,37 0,11 623,84 0,08 07:00 12594,37 0 623,84 0 07:20 12594,45 0,08 623,92 0,08

Between 6 and 6:25, the source sensor has 0 value, not even 0 (null for me) although the utility meter has the same value, no 0 or null (HA must fill the gap). When checking the Energy dashbord, I get the same delta values, 0,21 for the source sensor and 0,08 for the utility meter. I get why I'm missing 0,1 since maybe I got an issue when reading the sensor values. However not sure why I'm missing 0,03 though...

Matssa56 commented 10 months ago

Ok so I checked my esp code again and I'm not using the esphome api, I'm directly connecting to the mqtt broker in order to gain electric consumption (saw a website where the guy wanted to reduce the consumption to a minimum). Also, I send a online status with retain without sending an offline before going to sleep (to avoid showing the sensor unavailable). The issue can maybe come from there? Anyways I commented that line and the birth/will message and see how it goes tonight. I might need to use the esp wipe stuff to clear the messages.

Matssa56 commented 10 months ago

So changing the retain to the status had an impact though it might not be the best, the sensor is now offline and the energy dashboard is complaining, we'll see how it goes.

Matssa56 commented 10 months ago

This morning same issue, got a difference of 0.9 between the two sensors, even when putting it to unavailable when in sleep. The history graph of the source sensor is quite different now, it has only dots of data with null values in between (when the device goes to sleep) and the utility meter has the same thing now (unavailable when source sensor is unavailable).

dgomes commented 10 months ago

utility meter has the same thing now (unavailable when source sensor is unavailable).

That's expected behaviour.

Something I'm not understanding, please correct my assumptions:

Your ESPHome device has an energy sensor, which you feed directly to energy dashboard
You created a utility_meter with that same esp home energy sensor with which you are seeing differences

Matssa56 commented 10 months ago

utility meter has the same thing now (unavailable when source sensor is unavailable).

That's expected behaviour.

Something I'm not understanding, please correct my assumptions:

Your ESPHome device has an energy sensor, which you feed directly to energy dashboard

You created a utility_meter with that same esp home energy sensor with which you are seeing differences

Yup you got it ;-) the idea is to have peak and off-peak values for post-treatment and replace my energy sensor in my dashboard, that way I will be able to see what contract I need with my electrical supplier, check if solar panels are a thing, etc...

dgomes commented 10 months ago

There is a difference in the way energy dashboard handles calculations and utility_meter does.

utility meter calculates the diference between consecutive readings and sums all those diferences.

Your issue arrises from the fact that when the utility meter doesn't have 2 consecutive readings (because the source was unavailable) it will skip some of those diferences.

That's why I directed you towards https://www.home-assistant.io/integrations/utility_meter#periodically-resetting because it would take into consideration only the last good values (skipping the times the sensor is unavailable)

From my understanding, this option should have addressed your use-case...

Can you create a template sensor that uses the esphome energy value, and have the utility_meter use that template as a source ? Then share the graph of the 3 (esphome + template + utility_meter)

Matssa56 commented 10 months ago

There is a difference in the way energy dashboard handles calculations and utility_meter does.

utility meter calculates the diference between consecutive readings and sums all those diferences.

Your issue arrises from the fact that when the utility meter doesn't have 2 consecutive readings (because the source was unavailable) it will skip some of those diferences.

That's why I directed you towards https://www.home-assistant.io/integrations/utility_meter#periodically-resetting because it would take into consideration only the last good values (skipping the times the sensor is unavailable)

From my understanding, this option should have addressed your use-case...

Can you create a template sensor that uses the esphome energy value, and have the utility_meter use that template as a source ? Then share the graph of the 3 (esphome + template + utility_meter)

Hey, sorry for the late answser, had a massive storm hit France and lost my electricity, 4G, internet... Good days... ^^' Anyways so I reverted my MQTT sensor stuff to be as is as before (no will or birth message but mqtt message to say that it's online so that even when in sleep mode my HA sensor is available). This difference didn't change anything through my testing.

I created the following template sensor : {{ states("sensor.esp_linky_energie_totale") }} kWh, energy, total increasing

When checking the values near the reboot, both seem to be good however the template stops recording before the main sensor (6:02:18 vs 6:03:12) and gets the good value after the main sensor appears (6:04:01 VS 6:03:16) but these differences aren't that big.

However the utility meter still has issues since the value is still out of tune. Here are the values between 6 and 7 : Source sensor delta template sensor delta Utility meter delta 06:00 12672,76 12672,76 655,42
07:00 12673,77 1,01 12673,77 1,01 655,68 0,26

Here are some graphs showing the value change between these two hours : Utility meter

Sensors :

As you can see, the utlity meter doesn't take the value between before the reboot and after the reboot until the next change and confirms your saying. I can test again the periodicaly reset using the template sensor and see if i changes.

Matssa56 commented 9 months ago

Hey, just fyi the issue is still here and I don't know how to fix it. Maybe with a force sensor refresh after the reboot?

Matssa56 commented 7 months ago

Hi! Just checking if we can fix this issue since it completely breaks my energy calculator. Thanks!

issue-triage-workflows[bot] commented 4 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core