Incorrect values in Utilily Meters after Restart

[marc965]

The problem

I don't know the exact cause of this problem. As soon as I restart Home Assistant, all the values in the statistics are incorrect for the next hour. This is particularly noticeable on the Energy Dashboard. The recorder and the 'Utility Meters' seem to reset to 0 instead of using the last saved value. This began with Homeassistant version 2024.x

Without a restart, everything runs perfectly. The problem is triggered with every restart. I have already tried to reduce the load by decreasing the number of recorded entities, but unfortunately, this hasn't helped.

I have also tried to improve database performance by switching to an external database. The migration to MariaDB 10 was very successful, but it did not solve the problem.

Tried to reduce purge_keep_days. Setting: commit_interval: 30 Activating: auto_purge: true Acvtivating: auto_repack: true But nothing helps :-(

I started integrating my Victron Cerbo-GX device last year. Could this be caused by an increasingly high MQTT load?

This behavior is very annoying, as the statistics for an entire day are regularly completely ruined. It's a shame that there's no one there to analyze the problem. I am happy to offer my support.

Is probably linked with MQTT sensors or something else... Thats why i put this issue under label: "MQTT".

What version of Home Assistant Core has the issue?

core-2024.8.3

What was the last working version of Home Assistant Core?

core-2023.x

What type of installation are you running?

Home Assistant OS

Integration causing the issue

MQTT

Link to integration documentation on our website

https://www.home-assistant.io/integrations/sensor.mqtt/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

Here you can see that I restarted Home Assistant three times:

A day before - all statistics are ok:

[home-assistant[bot]]

Hey there @emontnemery, @jbouwh, @bdraco, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `mqtt` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign mqtt` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

mqtt documentation mqtt source _{^{(message by IssueLinks)}}

[jbouwh]

Seems a duplicate of #124167

[home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (utility_meter) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `utility_meter` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign utility_meter` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

utility_meter documentation utility_meter source _{^{(message by IssueLinks)}}

[marc965]

This morning i restart homeassistant and it happened again:

[dgomes]

please download the diagnostic of the utility meter

[marc965]

I am concerned that the Utility Meters created under 'Device & Services/Helpers' are working correctly. The issue is simply with the values displayed in the Energy Dashboard, which are incorrect after a restart. I do not know where these 'internal' Energy Dashboard relatet Utility Meters are stored.

For exampe, my own daily Utility Meter ist working correctly:

Where can i download the diagnotic of the Energy Dashboard relatet Utility Meters?

[home-assistant[bot]]

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (energy) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `energy` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign energy` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

energy documentation energy source _{^{(message by IssueLinks)}}

[dgomes]

can you share from the developer tools the state of the mqtt sensor ?

[marc965]

can you share from the developer tools the state of the mqtt sensor ?

[marc965]

[marc965]

[karwosts]

I do not know where these 'internal' Energy Dashboard relatet Utility Meters are stored.

Note the energy dashboard does not store any data. It only reads the data present in long term statistics for the given entities.

If it is showing big negative values, then something is causing recorder/statistics to reset it's internal sum count. But that's not something managed by Energy integration, that's managed by recorder.

[karwosts]

For example please make a statistics graph card on your dashboard for the "total energy to grid" entity, and especially plot the sum property. You will likely see a big discontinuity at the same time where energy is reporting these erroneous values.

[dhover]

I also have a problem with incorrect values after a restart of HA. Example: I have a solar energy sensor which is total_increasing. The start value of the solar for today is: 20188.876 kWh. The momentary value is 20190.587 kWh. Difference is 1.711 kWh. The solar sensor which is holding the solar energy is also used in a utility_meter with a daily reset. That value shows 1,706 kWh. Before the restart there was no difference between the values. Everytime I restart HA I am missing a small part of the solar energy in the utility_meter sensor. I have the same difference in other utility_meter sensors. It looks like the utility_meter is missing the increments during the restart of HA. Is this normal behavior?

[dgomes]

@dhover this issue is not about utility_meter (as reported by @marc965 )

Please open a new issue.

Setup debugging of the utility_meter and include the diagnostics of the misbehaving utility_mter

[dhover]

OK, How to setup the debugging the utility_meter?

[dgomes]

In the integration setting click the last button

[dhover]

Oh, that's new for me. Thanks for that.

[dhover]

I think I solved my problem. I had the option "Periodically resetting" enabled. see #88446. I disabled it and restarted a few times and now the utility_meter values are ok.

[dhover]

@dgomes Last question: I have a number of utility_meters defined. They are shown as a helper in the gui. Your screen dump shows them in the integration section. How is that possible?

[dgomes]

That is unrelated to issue, please discuss this in community forum and discord

That said, I don't know why 🙃 not a frontend developer myself

[marc965]

I do not know where these 'internal' Energy Dashboard relatet Utility Meters are stored.

Note the energy dashboard does not store any data. It only reads the data present in long term statistics for the given entities.

If it is showing big negative values, then something is causing recorder/statistics to reset it's internal sum count. But that's not something managed by Energy integration, that's managed by recorder.

The negative values are normal. When hey are negative this means thats feed-in... But the value is too high.

[marc965]

New Examples for today: All statistics before restart of HA are ok:

But after restart i have feed in over 35 MWh per hour:

Same with Solar Production. Before HA restarts the Solar production is normal:

Aber Restart i produced 156 kWh in one hour:

Statistics are completely useless. What can I contribute to speed up the solution to this problem?

[marc965]

i today analyzed my restart of HA around 13:30 based on Sun Energy Production. These Inputs are send via MQTT.

Here is my whole day sun production in energy board:

We see a huge peak at 14:00, because of my restart at 13:30. the sun production for every hour this day was: 0.28 kWh at 07:00 1.55 kWh at 08:00 3.42 kWh at 09:00 5.35 kWh at 10:00 7.33 kWh at 11:00 8.75 kWh at 12:00 10.04 kWh at 13:00

at 14:00 we see a large peak which could be the summation of all energy since 07:00. 0.28 + 1.55 + 3.42 + 5.35 + 7.33 + 8.75 + 10.04 = 36.72 So i assumed at 14:00 the energy production could be 48.94 -36.72 = 12.22 kWh

Conclusion: the energy statistics is resetting for the day counters and is beginning like the day has began. unfortunately the energy statistcs of this energy dashbaoard is not accessible. My own Utility Meters are working very well: you dont see any resetting, the graph is continously growing ant resetting at the end of the day.

This problem is the same for all energies that are coming from MQTT.

The Watermeter itself has no problems with that:

[jbouwh]

The values it self never come from MQTT, an other integration pushes the value to the MQTT broker. Most common problem is that the value is 0 while the sensor is unavailable. This will disturb the recorder when the sensor is total_increasing as it is assumed that the sensor was reset. If such a case can happen, an integration can use a value template to return None as a value for the sensor. This will make the state unknown. If the value is parses from a JSON, a null value will render to None.

[marc965]

The values it self never come from MQTT, an other integration pushes the value to the MQTT broker. Most common problem is that the value is 0 while the sensor is unavailable. This will disturb the recorder when the sensor is total_increasing as it is assumed that the sensor was reset. If such a case can happen, an integration can use a value template to return None as a value for the sensor. This will make the state unknown. If the value is parses from a JSON, a null value will render to None.

Hi @jbouwh. Could it be a solution to configure for my numerous Victron MQTT Sensors the payload_available, payload_not_available, availability_mode, availability_template and availability_topic options?

Numerous MQTT sensors are configured. The Victron sensors are not provided via an integration. At the moment, I haven't used an option like payload_available, payload_not_available, availability_mode, availability_template or availability_topic yet.

Is there anything else I can do so that after restarting Home Assistant the values don't default to 0, but instead take on a value that doesn't confuse the recorder?

Still, I don't understand one thing: If the recorder is confused by incorrect zero values, why is my total_increasing Utility_Meter sensor unaffected? It should also pick up the reset to 0, since it's receiving data from the same MQTT sensor, right?

[jbouwh]

That is certainly an option, but this is not the place to request for this type of support. Please use the Home Assistant Community for this. A project using these techniques is Zigbee2Mqtt. Using an availability topic is a way to control the availability of your sensor, but that is mainly used to control the availability of the sensors device it self. Using a value template and filtering out 0 is a solution if the sensor reports 0 when it is not available. The utility sensor is something different.

[marc965]

That is certainly an option, but this is not the place to request for this type of support. Please use the Home Assistant Community for this. A project using these techniques is Zigbee2Mqtt. Using an availability topic is a way to control the availability of your sensor, but that is mainly used to control the availability of the sensors device it self. Using a value template and filtering out 0 is a solution if the sensor reports 0 when it is not available. The utility sensor is something different.

I am sorry but i don't request any type of support. That was just an attempt to contribute to this issue or find a workaround. However, there's no need to be rude, we're all in the same boat and just trying to help...

Sorry for making a suggestion.

[schoppfe]

I just happened to come across this problem, which I am also experiencing.

I'm using a combination of AI-on-the-edge device with the MQTT Mosquitto broker and Home Assistant. I have already asked for help on the AI-one-the-edge page. In my case, restarting Home results in a single value for measured value and cost of zero, very similar to yours. The second value after restarting is correct again. As a result, however, the calculated costs jump up by the last known total costs.

https://github.com/jomjol/AI-on-the-edge-device/discussions/3278

I tried to provoke the problem by rebooting the external device (see gaps in sensor.waltermeter_value) but I can trace the problem back to the restart of home assistant.

[dgomes]

@schoppfe just use a filter https://www.home-assistant.io/integrations/filter/

[schoppfe]

@schoppfe just use a filter https://www.home-assistant.io/integrations/filter/

I assume you mean the outlier filter? Wouldn't that be like prescribing painkillers to a sick patient instead of diagnosing and treating the actual problem?

[dgomes]

@schoppfe just use a filter https://www.home-assistant.io/integrations/filter/

I assume you mean the outlier filter? Wouldn't that be like prescribing painkillers to a sick patient instead of diagnosing and treating the actual problem?

You have a problem caused by an external device/software (AI-on-the-edge), I give you a solution in HA that addresses these devices issues, and I'm to blame ?!

The filter integration exists for these situation! It's not like HA decided to drop to zero your value...

[schoppfe]

@schoppfe just use a filter https://www.home-assistant.io/integrations/filter/

I assume you mean the outlier filter? Wouldn't that be like prescribing painkillers to a sick patient instead of diagnosing and treating the actual problem?

You have a problem caused by an external device/software (AI-on-the-edge), I give you a solution in HA that addresses these devices issues, and I'm to blame ?!

The filter integration exists for these situation! It's not like HA decided to drop to zero your value...

Can we all please calm down again? No need for exclamation marks. I never said you were responsible for the error. I just don't like your approach. Even if it seems to lead to the goal more quickly at this particular point, I would still like to understand who or what caused the error and how. After all, it is not just this one entity that is affected, but the entire device. I don't know whether other entities of the device are being used elsewhere at this point in time or at a later point in time and are leading to errors either unnoticed or in the future. And that similar devices could be affected in the same way has not even been considered.

An update for HA just arrived and forced a restart. Good opportunity to observe the phenomenon again. No incorrect values are sent or received. Rather, the “not available” status is corrupting the statistics. Unfortunately, I still have no explanation as to where the two seemingly arbitrary values before and after restarting HA come from.

By the way. On the Ai-on-the-edge front, the error will almost certainly be blamed on Home Assistant, as the error only occurs after HA was unavailable and the device ran error-free.