Wrong data in the morning energydashboard

[mwitlox]

Describe the bug Wrong data in the morning energydashboard

Config Solarman portal config m.ginlong.com (Solis inverter, Wifi dongle)

Versions HA Version 2022.2.6 HACS version 1.22.0 Integration version v2.0.7

Debug logs < Will do this when needed Add to configuration.yaml:

logger:
  default: warning
  logs:
    custom_components.solis: debug

And attach the resulting log to the issue. If yo do not know where to find logs you missed an important step in the install of HACS. ;-)

Screenshots

14-02-2022 Time 8:54:23 7,3 kWh (from Yesterday 13-02-2022). Time 8:56:25 reset to 0 kWh.

Seems like ginlong is resetting this elsewhere. Buggy portal?

[bckmrk]

Hi, See #91

[mwitlox]

Hi, See #91

I know, i made #91. But the issue is back. Didn't find a reopen possibility so new issue made :)

[hultenvp]

🤦‍♂️ Shoot

[hultenvp]

Can you share sensor history of solis state and enery today?

[mwitlox]

Entity was off because i don't use it. Now it's on. I will wait until it happens again then make some screenshots :)!

I also have another test environment with your script but then connected with the Solis Cloud. I requested the needed info and configured HA. Curious what will happen there. Just to compare the issue. Don't know if they will stop using m.ginlong and migrate everyone to soliscloud one day..

Also asked a question about the firmware of my wifi stick because remote server b is unusable. Didn't get any firmware or fix for that they just said remote server b is in my case not usable. Still looking for a way to get local information. Don't really like this Chinese stuff

[LucidityCrash]

Also asked a question about the firmware of my wifi stick because remote server b is unusable. Didn't get any firmware or fix for that they just said remote server b is in my case not usable. Still looking for a way to get local information. Don't really like this Chinese stuff

I did the same thing ... got the same answer ... pointed out that other people have been doing it, and sent links to the pages ... got silence.

[bckmrk]

I have a really strange thing going on in my energy dash yesterday... dont know if its related to this issue though? two peaks of energy reported, one between 8-9 and one 23-00... the peaks is almost consistent with my total consumption at this time this day. And no, the sun did not shine between 23-00, as indicated by the purple field...

The sensor historiy doesn't look this at all so I don't know where energydash gets this data from. Is it related? Probarbly not.

[michaelsargent]

I think the bug may be in the energy dashboard as I get an outlier from an energy meter on random days.

[hultenvp]

Also asked a question about the firmware of my wifi stick because remote server b is unusable. Didn't get any firmware or fix for that they just said remote server b is in my case not usable. Still looking for a way to get local information. Don't really like this Chinese stuff

I got two brands of inverters, Solis and Omnik, both use same type of stick, different form factor, so I assume they get it from an external supplier. Omnik is fine adding server b and I use that to directly read from inverter, solis is indeed not supporting server b. Never found a FW update

[hultenvp]

I think the bug may be in the energy dashboard as I get an outlier from an energy meter on random days.

Could be that it is a bug in the dashboard, but the only way to be sure is inverter energy today and state history. Until now all peaks on the dashboard were also visible in the history. If you see peaks in the circled area then it's the integration still not fully filtering out false data from the day before.

[bckmrk]

As I wrote earlier there is no peaks in my sensor data (your circled area)... thats why I thought it was not related to this. But still wierd. :-)

[hultenvp]

@bckmrk: Had some more time to look at your issue. As you already stated the peaks are in consumption, didn't realize this at first. Do you get that grid data from a smart meter, from the grid data from the Solis integration or something else?

[mwitlox]

@hultenvp Nothing strange happend yet.. but i was thinking. The Energy Dash, this gets information every hour i think? So the value from yesterday is there when the inverter kicks in the power from the grid. After that kick in m.ginlong resets complete and starts counting from 0. Timing issue with the dash? Bad software at m.ginlong? The kick in was almost 9 o'clock 8:54 - 8:56 (Screenshot 14-02). Hard to filter this out :(

My second HA with the other way SolisCloud. Just a small thing i wanted to share. Don't know if its related to the soliscloud. Started 2022-02-17 21:14:58

I can make another call and share the info for improvements?

Also still waiting for the m.ginlong issue coming back, then compare with the soliscloud.com

[LucidityCrash]

the SolisCloud API errors were probablg due to the API outage I reported in the beta discussion thread ... started a bit after 09:00 UTC on the 17th and was fixed about 01:00 utc on the 18th don't kjow your TZ so may not be that

[hultenvp]

Hi @mwitlox,

As @LucidityCrash already indicated the retries are because of maintenance downtime on SolisCloud. The integration will continue to try to connect until it is able to successfully connect to the server. So what you see is as designed ;-)

Regarding the ginglong.com: Are we talking about the energytoday sensor or griddailyongridenergy (or one of the other grid sensors)? Until now I was assuming it's the energytoday sensor where you still see spikes, but I'm getting the impression that might not be the case. If it is energytoday where you still see the spike then my filter still does not cover all conditions and then I really need more info if it happens again. If it is a grid sensor then we have the cause, because the filter does not work on that sensor yet (my inverter does not provide grid data, so don't know if also grid is reset at the wrong moment).

[bckmrk]

I get the grid data from a live meter connected to my electricity meter. It’s not a smart meter but something else… it’s a Tibber pulse. I restarted HA several times during the hours when the peaks I had the other day occurred but have never seen that it causes spikes before.

[mwitlox]

@hultenvp

Thanks for the SolisCloud info, didn't know about the maintenance.

I made some new screenshots. Last day it happend was 14th (Running version 2.0.7) . All used sensors should be visible. All from the integration.

Same grid.

1. m.ginlong

2. SolisCloud (Active since 15th for testing)

[hultenvp]

Thanks for the data, @mwitlox,

Bit odd that the ginglong inverter state y-axis does not match the selected time period, seems to start Feb 15th afternoon. Means I can't see if there were hiccups in the state data around the time of the spike event on the 14th. Could you try to dig up the state data around the spike event?

Also interesting to see that the energy today on the soliscloud beta integration resets before the inverter state changes. It already resets at before 18:00, where the state changes to off (2) after 18:00. Is not going to negatively affect energy dashboard, but still...

Another question: when you start using Soliscloud, did the data get migrated from ginlong.com, and if so, was the historical data on the ginlong portal deleted as part of the process? I need to start using SolisCloud myself as well, but would like to get a better picture what to expect.

Cheers

[LucidityCrash]

Also interesting to see that the energy today on the soliscloud beta integration resets before the inverter state changes. It already resets at before 18:00

I'd notice and @enrique had also seen the energy_today resets at 16:00 UTC (enrique's reset at 10:00 UTC-6)

[bckmrk]

Hey, another issue.. maybe related? Dont know if the error is within HA or within Solis integration but upon HA-restart the energy dashboard data from Solis-integration "sensor.solis_energy_today" adds its current value (again) to that hour.

Example: 1:st march I produced a total of 17,7kWh but energy dashboard says 20,1 - I restarted HA at 9:56 AM (when todays total were at 2,4kWh), back online at 10:10, (!) very long restart , when todays total were 2,7kWh. The difference 20,1 - 17,7 is 2,4, as todays total was when I restarted.

Is it just me? :-)

[LucidityCrash]

I've not noticed that but to me it looks like a HA Energy Dashboard bug ... look at your 2nd picture, that is the raw data being collected the graph in the first picture is just another way of showing the data ... at a simplistic level it takes the value at the begining of an hour window and then again at the end and then displays the difference between the 2 as a Bar. This is nothing to do with this integration - at least I don't believe so 😁

[bckmrk]

… yeah. But how about my other two sensors linked to energy dash that doesn’t show any sign from the restart, with a continuous line from the sensor. Can this be achieved with the solis integration as well? Or what’s the difference?

“Exported” sensor:

/ curious and not that in to how everything works… (-:

[LucidityCrash]

Now I say it out loud again .. I think I see the problem ... as you point out the issue is that flatspot that drops to 0 ... it is tripping up the calculations for the delta of that hour. Personally I've no idea how to stop the flat spot though

[LucidityCrash]

and I can confirm this behavior too today I restarted HA have a flat spot and have a spike in the orange bar and a daily yield that is higher than reported by Solis. (this is using the SolisCloud API). I'd say the root issue is how HA does its calcs BUT the problem is created by this integration dropping the yield to 0 after a restart. Trouble is I think this might be a side effect of the workaround to stop yesterdays value being seen on the initial startup of the inverter in the morning before things reset.

[hultenvp]

Hmpf,

Damn if you do, damn if you don't. ;-) Let me see if I can add robustness against reboots in the morning without breaking the energy today patch or having to introduce persistency. Maybe returning "none" instead of "0" is the way out.

In any case thanks both for zooming in on the root cause, this really helps.

[hultenvp]

In service.py line 157 & 158:

                            elif last_updated_state + timedelta(minutes=10) > datetime.now():
                                value = 0

If you replace that with:

                            elif last_updated_state + timedelta(minutes=5) > datetime.now():
                                return

Does that help?

This would skip updating energy today instead of reporting value 0. In principle this should stop the energy dashboard reporting jumps. I'll also try to reproduce here.

[LucidityCrash]

I shall give it a try.

[LucidityCrash]

So after that patch and a restart of HA my Energy today graph looks like : and the energy page looks like :

which seems to fit in with all the data ... no double counting of the yield at the point of restart.

[LucidityCrash]

the thing that is interesting though is that it would appear that while the sensor is just doing the return HA graphs that as the last known value. The gap is only a minute so the 10 minutes till the next bump are 5 mins from the code and the fact that it wasn't really generating much as it is a horribad day 😬 Doesn't that mean that it won't fix the problem that the return 0 was put there for in the first place

[hultenvp]

note that the energy today patch only works at AM, so testing in the afternoon will always report correct values. 😛

The patch has two cases it covers: 1) If it's AM and the inverter is off --> Return 0 kWh for Energy Today 2) If it's AM and the inverter is on --> Return 0 kWh for the first 10 minutes after switching on to avoid still capturing an old energy today version. Step 2 also triggers when you reboot in the morning as it considers the first state measurement after restart as the moment of last measurement. This means that for 10 minutes after restart you'll see 0 kWh reported.

Step 2 is now changed into --> Don't report anything for the first 5 minutes after switching on. HA will consider the last recorded value as still valid.

[LucidityCrash]

doh ... I missed that ... will restart again in the morning the next time I remember :)

[hultenvp]

At least for me it seems to have the desired effect. Interested in your results still. The final fix will look a bit different though. It works, but breaks the for-loop

[bckmrk]

Looks like this after the change - problem solved? :-) No double counting in energy dash.

[LucidityCrash]

I concur ... looks good to me.

[hultenvp]

The final fix is in master, will be in next release.

[LucidityCrash]

NICE 😁

[hultenvp]

Waiting for confirmation on issue #93, then I'll release.

[mwitlox]

I need to start using SolisCloud myself as well, but would like to get a better picture what to expect.

Sorry for my super late response. Alot going on here. Second time Corona. Vaccin didn't help, got more sick this time then without vaccination.. Now i'm ok again!

The inverter state was disabled before the 15th so i cannot see the data when the hiccups happend, sorry

About the soliscloud seems like it went dark almost every day around the same time. Here a screenshot from this week And here from today for some more detail

About the migration, i didn't lose any data! When i first logon to the new portal all data was migrated automatically without issues. Don't know if there are users that had any problem with this.. Still both apps are working fine.

All days after i reported this issue it didn't happen again. No wrong data, all fine. Still if you like to look at more data we can do this together some time? Remote session maybe. If i can help improving the scripts by providing more info happy to help out! Just don't ask me much about programming haha

[hultenvp]

Same here, no covid, but still extremely busy so little time for fun stuff.

Good to hear no wrong data occurred. I'm going to drop a new version today with the fix for @bckmrk 's problem. I also think it might somehow have impacted you. If there's still a problem with wrong data then just create a new issue. In that case I'd be happy to plan a session to gain better insights in the data.

By the way, if you look carefully at the state data then you see it shifting a bit through the week. Nice to see days are getting longer. +1 kWh extra if I compare my PV production from yesterday with 1 week ago. 🌞

[hultenvp]

Released in 2.1.0