Lifetime Energy report 0 sometimes

[jonas-holm]

The problem

Some days sensor.solaredge_lifetime_energy reports 0 and the energy graph show incorrect values. I get -35000 kWh and then when the server connection is back after a few hours it shows +35000 kWh. Would it be possible to create a function that ignores decreased values on these sensors as this allways should increase or stay at the same value to stop problems like this when the SolarEdge servers connection is lost for som reason. If the value is less than the reading before just set it to the same value as the last sensor reading. Have seen that other integrations have the same issue and this would solve there problems to o.

What version of Home Assistant Core has the issue?

2021.11.1

What was the last working version of Home Assistant Core?

none

What type of installation are you running?

Home Assistant OS

Integration causing the issue

SolarEdge

Link to integration documentation on our website

https://www.home-assistant.io/integrations/solaredge/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant[bot]]

solaredge documentation solaredge source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @frenck, mind taking a look at this issue as it has been labeled with an integration (solaredge) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[jonas-holm]

I think this is not actually a problem with Solaredge integration rather the energy statistics and how they are gathered and processed. Seems like more solar integrations have the same issue. From what I understand about the problem you could just compare the lifetime energy sensor reading with the last reading one hour ago and if it is lower it is wrong as a lifetime energy sensor can't decrease (you can't have negative power production) and then just set it to the same as last hours reading. It will ve wrong but will then correct itself when the cloud service is up again. If it happens at night which it usually does it won't even be wrong.

[liudger]

I think this is not actually a problem with Solaredge integration rather the energy statistics and how they are gathered and processed. Seems like more solar integrations have the same issue. From what I understand about the problem you could just compare the lifetime energy sensor reading with the last reading one hour ago and if it is lower it is wrong as a lifetime energy sensor can't decrease (you can't have negative power production) and then just set it to the same as last hours reading. It will ve wrong but will then correct itself when the cloud service is up again. If it happens at night which it usually does it won't even be wrong.

I think it is a statistics problem. If some sensor give temporary some wrong data the calculations should correct this. Maybe statistics should do a check once a day see if the results the are calculated in short term statistics had some errors. And correct them. https://github.com/home-assistant/core/issues/57900

[ChristophCaina]

same as #57045

the problem here is, that the integration (as well as the statistic calculation) can't really check, if the value of 0.0 really is a valid information or not. Maybe, it would be possible, if there's a way to do some code rework and check the sensors value together with log messages - for example, IF the integration does receive an 429 error (connection limit reached) or an 404 (site not available, etc) it could react somehow on the 0.0 value witch will result from this.

[jonas-holm]

It should be able to check if the sensor value is less than the last value. If so its wrong as a "life time" value can never decrease. You can't have negative solar production.

[ChristophCaina]

as far as I got from @frenck , the LifeTimeEnergy sensor from SolarEdge can return a slightly smaller value sometimes. This is usally being caused by some watts the inverter is consuming during shutdown or wakeup / at least over night.

As far as I understood, it might be that the lifetime sensor does reply at least a slightly smaller value on the next morning than on the evening before.

@frenck , am I right?

[jonas-holm]

Everytime I get this problem it reports 0 kWh and the shifts back to the correct value after one or two hours. An easy fix seems to be to ignore 0 as Life Time Energy and also ignore large decreases (>2 kWh perhaps) to fix this problem. The Inverter will not draw that much power during the night.

Could this be a viable soution @frenck ?

[JaccoR]

By doing this, you would ignore 2 datapoints, 1 each hour, and therefore the graph would not be accurate anymore, right? btw, is the 0 value coming from solaredge API because of an error there, or is the 0 value there because of an error in home assistant?

[liudger]

Here are my thoughts about how to tackle this issue with wrong data.

Statistics is the best way to get these wrong values out. There are also other integrations that gives wrong data. So or we need to fix all those integrations (we should) but the better approach is with statistics to cancel out these outliers off wrong data. It's all for the enduser who doesn't understand integrations and how to fix them and who doesn't understand going into a database to correct data. There will be always integrations that send wrong data to the statistics. Statistics shouldn't be unusable if this happens. So we need some self healing power!!

So I would strongly suggest that we tackle these kind of errors within the statistics generation. Maybe we can look into some current solutions for generating some clean statistics data? There should be some libraries where we can check data points and correct them. I am no expert on this yet.

https://github.com/yzhao062/Pyod

https://www.kaggle.com/taruntiwarihp/outlier-detection-in-python-using-pyod-library https://www.geeksforgeeks.org/detect-and-remove-the-outliers-using-python/

[jonas-holm]

By doing this, you would ignore 2 datapoints, 1 each hour, and therefore the graph would not be accurate anymore, right? btw, is the 0 value coming from solaredge API because of an error there, or is the 0 value there because of an error in home assistant?

The graph wouldn't be accurate but more correct than now when it first reports a negative production and then after one hour a positive production that actually is the total lifetime production.

I think the error comes from SolarEdge when you get no respons for some reason. Maybe as a result of to many api requests.

[JaccoR]

Yes that is true.

I get the error of too many requests sometimes and this does not give a 0 value, just no data at all. It happens for some other reason.

Btw, if other solar integrations have the same problem, how are they fixing it?

[jonas-holm]

Yes that is true.

I get the error of too many requests sometimes and this does not give a 0 value, just no data at all. It happens for some other reason.

Btw, if other solar integrations have the same problem, how are they fixing it?

My idea was that this would fix all solar integrations in one go.

[sienikam]

I have similar situation today.. https://imgur.com/a/Uyqfyha

anyone has workaround or fix for that situations? @jonas-holm ?

[jonas-holm]

I have similar situation today.. https://imgur.com/a/Uyqfyha

anyone has workaround or fix for that situations? @jonas-holm ?

No nothing yet. Still hoping for a fix everytime I update...

[bkbartk]

Same issue here. +1 for fixing this But I think the integration works correct. There is an error in the solaredge api. And the energydashboard should cancel out negative solar production.

[jonas-holm]

Hoping for some @frenck magic here... After talking to some people I have come to the conclusion that this problem exists even with other cloud based integrations and local (Solaredge local) integrations so it would be great to implement some failsafe function in energy stats to ignore negative values if for example an integration failes to get values. From what I understand the problem would be fixed if energy ignores the value 0 and that could be a clean and good working fix.

[bkbartk]

I found that the mySolarEdge app itself also uses the wrong value. It doesn’t show in the stats. But daily production was ok. All was ok. But this afternoon total production showed 0kwh. Where now it’s 1.03 mwh. so definitely the api. Still a fix would be nice.

[jonas-holm]

Yes it is a problem with the api but the energy stats should have a failsafe for this. I have friends who have the exact same problem with Solaredge Local when they restart HA because of startup times. If the energy stats ignore 0 values and in case of 0 just copy last value, this would be fixed sort of...

[humpedli]

I can confirm this issue on my side as well, mySolarEdge app also shows wrong value. A failsafe solution would be nice for this.

[xaastr0m]

I have the same problem and it has come more often lately

[jeroenhoebe]

I can confirm it's getting worse. Almost everyday now.

[liudger]

Yes we need to do something about this. +1 for some action you can run on the database to correct those values. Could be something simple.

[robsonke]

I face the same issue and correct it myself for now on the db. Just query the statistics and states tables for that period and entity. You'll notice a gap in measures, just populate them with the previous value.

[hofkickser]

Same issue here as well...

[bkbartk]

this query fixes the issue for me.

Delete FROM "statistics" where metadata_id = and State = 0.0

in de "SQLite Web" add on. this is not a permanent solution as it will only fix historic issues and not prevent it from failing in the future. but I want to share this simple query anyway.

Yes I know the query can be improved adding a Join, but I used the one above.

[sienikam]

Fix which works for me is to change from sensor.solaredge_lifetime_energy to sensor.solaredge_production_energy looks better for now..

https://community.home-assistant.io/t/solaredge-integration-negative-values-of-produced-energy/377462

[bkbartk]

I checked "sensor.solaredge_production_energy" this one seems to return the current generated amount of today. This might will work for overviews per day. but when showing week/month/year overviews does it still work?

[sienikam]

I checked "sensor.solaredge_production_energy" this one seems to return the current generated amount of today. This might will work for overviews per day. but when showing week/month/year overviews does it still work?

yes :)

[leandresz]

I have this same issue quite often.

Somehow the is a correction also. But the graphs don't look as neat as you want it to be :)

[liudger]

With sensor.solaredge_production_energy the caps will be less extreme. So the charts will be more usable. But when the sensor will be unavailable the wrong data is harder to detect because of the smaller delta? With the total energy the faulty data is easily spotted. Also because the total will correct itself the data in home-assistance will correct itself. See solar production chart. But if the daily energy production will reset itself every night it has no possibility to correct itself? So I am not sure if this is the better way to measure the energy.

But in your ha forum post you use a template. So will this solve the issue? Like when overnight the sensor is unavailable it will give the last cumulated template value of the sensor and when the sensor is available it will reset itself to 0 it will cause a negative value? So I am not sure if this is really a better solution. Maybe if the template has also an automatic reset at 00:00

Still I would prefer a solution with the energy integration that can catch those unavailable situations. This is because multiple sources have the same issue. So fixing this per solar integration is less ideal.

Also a fix button that looks over the statistics and suggest fixes!

[frank8the9tank]

I have exactly the same problem, values becomes zero.

[hofkickser]

I manually repaired the issue in the following way:

1. Find the appropriate statistics_id by querying the statistics_meta table using the SQLite Web add on. In my case it was id 26:

2. Run the following query to identify how often the issue occurs (change metadata_id by the number acquired from step 1): SELECT * FROM "statistics" where metadata_id = 26 and state = 0.0

3. In my case approx 70 records appeared to identify the issue. All records have a UTC timestamp of when the Solaredge API was unavailable resulting by column state 0.0 and a negative number for the column sum example:

4. Find the previous value (before the issue occured) and not the values of column state and sum. In my case the Solaredge API was unavailable on 2022-03-06 at 21:00 so I recorded the values of 20:00:

5. Update the incorrect record(s) of the database (21:00 & 22:00 hrs in my case using the following sql command: update statistics set state = '5910.449', sum = '1209.5459999999994' where id = '250521' ;

and repeat this sql command to update the database record with id 250587

6. The result looks like this:

Yes, it does take some time to figure out the correct values to update, but this will repair all daily, weekly and monthly graphs in the Hass energy dashboard. It took me 30- minutes to correct approx 70 values, but everything is working fine since then:

[ulmerkott]

I've observed that it's only the "lifeTimeData" that is set to 0.0 for the overview API-call. The other values, "lastYearData", "lastYearData", "lastMonthData" and "lastDayData" are still OK (non-zero).

This is typically how the data looks like when the problem is reproduced:

{
  "lastUpdateTime": "2022-03-10 20:26:08",
  "lifeTimeData": 0.0,
  "lastYearData": 1048586.0,
  "lastMonthData": 386887.0,
  "lastDayData": 18380.0,
  "currentPower": 0.0,
  "measuredBy": "INVERTER"
}

I created a pull request that aims to workaround this problem by skipping the update if "lifeTimeData" is less than last Year, Month or Day.

See https://github.com/home-assistant/core/pull/68048

[Frederic98]

I've been making some API calls by hand whenever HA shows wrong values for SolarEdge to try to see where the problem lies. For my instance, the Lifetime Energy doesn't go to 0.0kWh, but drops by 16.5-and-a-bit MWh. This means that the solution to this problem can't just ignore 0 values. After looking at the SolarEdge Monitoring website, it also shows the lower lifetime stats there.

Curiously, the yearly production for our installation is around 16 (+/- 1) MWh. So it's almost as if the server responds with lifetime statistics for a year ago.

[bkbartk]

That can explain the 0 in my case. A year ago I didn’t have solar panels yet.

so actually the value should be ignored when it’s lower then the previous one?

[PitinoB]

I repair 0 in my statistics table with this query

update statistics set state = (select s.state from (select from statistics where metadata_id = 9) s where s.id < (select id from statistics where state = 0 and metadata_id = 9 order by id limit 1) order by id desc limit 1), sum = (select s.sum from (select from statistics where metadata_id = 9) s where s.id < (select id from statistics where state = 0 and metadata_id = 9 order by id limit 1) order by id desc limit 1) where state = 0 and metadata_id = 9;

metadata_id is my sensor id from metadata table

[liudger]

They did fix statistic short and longterm. (so no more wrong values to correct) So with this pull request coming it will be solved https://github.com/home-assistant/core/pull/68048 Only the zero values are there and those are easy to fix.

check the sum 0 values

SELECT *
FROM "statistics"
WHERE metadata_id = 'id of your statistics to fix'
AND sum = 0

delete them

DELETE
FROM "statistics"
WHERE metadata_id = 'id of your statistics to fix'
AND sum = 0

[JohnnyM84]

i'm trying this with phpmyadmin because i use mariadb addon.

if i use the query SELECT * FROM statistics WHERE metadata_id = id AND sum = 0

Then i see only entries for the first 2 days of when i created the database (87 rules), but not for all days. does someone know how to do this for every row in statistics and not the first 2 day of rows.

Edit: problem for me was not in the sum 0 but in state 0. Solved it with: SELECT * FROM statistics WHERE metadata_id = id AND state = 0 Delete FROM statistics WHERE metadata_id = id AND state = 0

Then i had 12 days were there was not a 0 problem but a problem of an increse and decrease of 1500+, i deleted this with: SELECT * statistics WHERE metadata_id = id AND sum > 1500; DELETE FROM statistics WHERE metadata_id = id AND sum > 1500;

now everything looks OK