IoTaWatt integration reporting incorrect values

[tomlut]

The problem

My solar inverter is only capable of outputting 6kW so imagine my surprise to see the core Home Assistant IoTaWatt integration reporting it producing 6.7kW:

The IoTaWatt itself never recorded a value this high:

What version of Home Assistant Core has the issue?

core-2024.4.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

IoTaWatt

Link to integration documentation on our website

https://www.home-assistant.io/integrations/iotawatt/

Diagnostics information

Not available

Example YAML snippet

UI integration

Anything in the logs that might be useful for us?

Nothing related to IoTaWatt

Additional information

No response

[home-assistant[bot]]

Hey there @gtdiehl, @jyavenard, mind taking a look at this issue as it has been labeled with an integration (iotawatt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `iotawatt` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign iotawatt` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

iotawatt documentation iotawatt source _{^{(message by IssueLinks)}}

[lyricnz]

That's interesting. I don't know how this could happen - iotawatt component just uses the REST api to talk to the iotawatt device, and stores what it gets. Can you try zooming in on iotawatt graph, see if there was a spike around then? Maybe some interference?

Mine seem to align pretty well

[tomlut]

That's as close as I can zoom in. I can't zoom in any closer than a +/-5 minute period in the history panel.

[tomlut]

Also as it it past the 7 days I store in the recorder I only have 1 hour mean LTS data now.

[lyricnz]

@tomlut Is your solar_prod a direct measurement, or is it an output with a formula etc? Other than that - I have no idea: HA just uses the values directly from the REST interface to iotawatt.

[tomlut]

It is an IoTawatt integrator output.

[tomlut]

Just noticed it happened again a couple of days ago:

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[tomlut]

I have not seen this occur since my last report. Though nothing has been done to fix it. So I'll re-open if it occurs again.

[tomlut]

This is still occurring.

[tomlut]

This integration is still reporting incorrect values. My inverter is simply incapable of producing power over 6kW.

The IoTaWatt never records values this high yet the HA integration does:

[tomlut]

Still happening 2024.9.1

[boblemaire]

I don't have much context here, but I think I have an idea what is going on. @tomlut stated that the source of the data is an IoTaWatt integrator. As per the IoTaWatt docs, he is probably using energy production (import and export) in HA. As we all know, energy is expressed in Watt-Hours (Wh or KWh). Although the time period associated with Wh is always provided in IoTaWatt query responses, I think I recall that HA does not bother with that and simply timestamps data with time of arrival. As a result, its possible that HA is developing Watts by dividing Wh by potentially inaccurate hours (or fraction thereof). If the time period is understated, the resulting Watts will be overstated.

This is just one example of the deficiencies in the way HA energy approaches energy accounting. There are other flaws in their approach. Bottom line is that regardless of the nice graphics and moving arrows, the underlying data does not have historical integrity because it is subject to these timing inaccuracies, and more importantly does not have any way to back-fill when data is lost due to LAN or server issues.

When all this started, I implored HA to consider working with me to resolve these issues in the same way that IoTaWatt maintains other external data uploads. They had no interest. IoTaWatt can maintain perfect time-series energy data with influxDB, EmonCMS or PVoutput. They use the timestamps, and upload interruptions are backfilled from IoTaWatt's datalog upon resumption. HA is fine for a real-time power meter display, but it has no historical integrity. Note that the posts above compare the HA outputs to the gold-standard IoTaWatt Graph+ data.

My recommendation to anyone that wants an easy solution to historical energy data in a convenient app that can be accessed when not on the IoTaWatt's LAN is to use the free and easy PVoutput. IoTaWatt can maintain that and still feed HA. It takes just a few minutes to setup a free PVoutput account and start keeping accurate historical energy production and use data. Take a look at this site that has more than 5 years of virtually uninterrupted historical data as well as nice daily, weekly... yearly usage graphics.

[jyavenard]

I think I recall that HA does not bother with that and simply timestamps data with time of arrival. As a result, its possible that HA is developing Watts by dividing Wh by potentially inaccurate hours (or fraction thereof). If the time period is understated, the resulting Watts will be overstated.

The energy meter takes the accrued energy value from midnight. So it is accurate for the day, as accurate as what the iotawatt provides.

It used to integrate the power value over time but this changed as you added support for integration sensor.

Querying it's value from midnight has its challenges too, and you lose accuracy here. Originaly I made HA query energy from the beginning of the year so it was more accurate. Somehow made a PR to only grab the value from midnight and unfortunately it went in.

FWIW I no longer use the HA integration, it's too crippled by design choice forced by HA. Instead I rely on the REST integration and directly query the iotawatt over http.

[tomlut]

Hi Overeasy, nice to see your product expertise here but there is a little confusion.

This is a power sensor, not an energy sensor. It is an IoTaWatt power output generated from an integrator that only reports positive values as the inverter consumes power at night that I do not wish to record as negative solar production. This was the setup recommended on the IoTaWatt forum, https://community.iotawatt.com/t/enphase-solar-input-nighttime-power-consumption-workaround/1368/30?u=tom_l

The energy reported by this integrator to home assistant agrees perfectly with that recorded by the ioTaWatt and the monthly total is accurate to better than 2% compared to my energy provider's metering. Nice.

The power reported by the output never goes above 6kW in the IoTaWatt. Which is good as the inverter is simply not capable of this.

However the IoTaWatt power output recorded in Home Assistant sometimes exceeds this by a lot (7kW spikes). . These spikes are never recorded in the IoTaWatt so it has to be either the HA integration implementation or IoTaWatt API reporting that is at fault.

Solar Power Input (works well in both HA and IotaWatt, but reports consumption at night)

Positive Solar Energy Integrator (works perfectly in both to report positive energy)

Positive Power Output (works well in IoTawatt but has erroneous spikes in HA)

[tomlut]

It just occurred to me that for measuring positive solar power I could create another output that does not rely on the integrator. It will be interesting to see if this suffers from the same >6kW spike issue.

[boblemaire]

Hello @jyavenard, yes it's been awhile. As I said, I don't have a lot of context here. You know a lot more about the way HA is using this data. But we seem to agree on the fact that HA timestamps the data at the time it is received. That was my main point.

As I understand it, the integration is pulling cumulative Wh every 30 seconds (or so). So lets say that at 12:30:00 HA requests the cumulative Wh for 12:30:00 and the returned value is arbitrarily 10,500 Wh. Also, lets say there is a three second delay in the response. The integration saves the 10,500 Wh value and it is timestamped 12:30:03.

The inverter is maxed out and producing 6,000 Watts, which is 50 Wh of energy every 30 seconds, or 55Wh every 33 seconds.

At 12:30:33 the integration again queries the IoTaWatt for cumulative Wh as of 12:30:33. The response is 10,555 Wh. As there was negligible delay, that value is stored with a timestamp of 12:30:33.

So now the two values in the HA database are:

12:30:03 10,500 12:30:33 10,555

If HA were to calculate Watts for that interval it would divide Delta Wh = 55 by Delta time 30 seconds (0.00833 hours) = 6,602 Watts. This looks a lot like the problem that is being reported.

That was a simplistic example. Looking at the HA graph above for 31 August, the actual intervals are almost all slightly greater than 30 seconds, suggesting that the integration does not run periodically, but rather delays 30 seconds each time it has run, thus adding the transaction time into each interval. Nevertheless, the request time is probably the dispatch time and the timestamp is, as we agree, the response time. It's a shame, because the data from the IoTaWatt has a timestamp accurate to a few milliseconds.

Does this seem plausible?

[jyavenard]

Does this seem plausible?

No. HA is pulling every 30s the data for the amount of energy since midnight. It doesn't do any averaging. So the longer in the day, the more accurate it becomes as delays become proportionally smaller.

That's also just for energy. For power it pulls the power value.

All I can say is that the energy figure are correct. For power? :shrug: HA does weird stuff indeed when it restarts. It's like it's trying to fill up missing data with some unknown quantity which is typically rubbish.

So after all SW update I do see spikes. I have stopped using HA to compare my bills. It's just too buggy. And all my attempts to get thing accurate ended up being reverted for one reason or another. Usually by someone new at this who wanted a particular figure disregarding long term accuracy.

[tomlut]

That's also just for energy. For power it pulls the power value.

And this is the crux of the matter. Why is the power value reported by HA greater than what IoTaWatt has recorded in its database?

My energy totals are working as expected. There is no issue there.

I also went back and checked, I did not restart HA at the time the spikes occurred.

[jyavenard]

Enable debug logging, look at the time you get nonsensical value, check in the log what value was pulled.

[tomlut]

Ok, debug logging for IoTaWatt is now enabled. However the issue only happens infrequently so we may be waiting a while.

I also have two other sensors from IoTaWatt to compare it to. The solar input (that goes negative at night) and my new sensor mentioned above.

[tomlut]

Well we didn't have to wait long, there were two impossible spikes today:

Only the power sensor generated from the integrator had the spikes. The solar.pos output power sensor and the solar input (that goes negative at night, so I cant use it) did not.

None of the three solar power sensors registered this in the IoTaWatt:

The debug log gives nothing relevant for this time:

debug.txt

I suspect this is going to be impossible to solve so I will switch from using the solar.pos power sensor from the integrator to the solar max 0 sensor generated from the input for power while continuing to use the integrator for positive energy.

[jyavenard]

unfortunately here, we can't see what data was retrieved by HA from the IotaWatt or if one sensor became unavailable.

[tomlut]

None of the sensors became unavailable.

[tomlut]

I have a solution.

Changed this:

To this: