Midnight glitches in Current Month and Current Year data

[AndersHoglund]

The lifetime energy data has a fix to filter out the midnight glitches, but not the Current Year and Current Month data that also has those glitches. This issue is only as a note for the future, it is not a real problem in the standard energy dashboard. Only visible in history graphs. This is seen using Kiosk, not sure if it also exists when using OpenAPI.

[tijsverkoyen]

@AndersHoglund I think https://github.com/tijsverkoyen/HomeAssistant-FusionSolar/pull/142 should resolve this.

Would you be able to test/verify this? If you apply the changes on your local instance and be able to report back if the issue is resolved.

I'm really sorry that I'm not able to debug it more properly myself, but I don't have any glitches.

[AndersHoglund]

@tijsverkoyen I'm sorry but that will not work. Monthly and Yearly data will be cumulative forever, they need to be allowed to reset to zero and start over at day 1 in each Month and Year respectively. I.e. need to be handled differently from Lifetime Cumulative energy. Funny you can not see this. Are you using OpenAPI only, and no such problems there? Not sure this needs fixing. This data is normally not used in the HA Energy Dashboard.

[tijsverkoyen]

Hmm. Idd. I will check again.

[tijsverkoyen]

@AndersHoglund I have reverted that specific change.

But there is no check in the code that the does not allow the value to be lower that the previous one.

cumulativeEnergy will only update if the system is producing energy, which was already the case. I have improved the code that none float values will be handled better. See https://github.com/tijsverkoyen/HomeAssistant-FusionSolar/pull/142/files#diff-ef138f2cea7ca5ced551123ebe8e221e3d2d1a839beb5683367aced315e3b6acR90

Would you be able to test these changes?

If Huawei actually returns 0 the glitch will be present. The only other way is:

For monthEnergy: checking if the value is lower than before if so don't update the state. Except if it is the first of the month. But in that case the glitch can still appear on the first of the month.

For yearEnergy: checking if the value is lower than before if so don't update the state. Except if it is the first of the year. But in that case the glitch can still appear on the first of the year.

And this also means that if the API is down on the first of the month/year. The data will not be updated untill the next first of the month/year. So introducing a lot of problems in my opinion.

The best solution would be that Huawei fixes the data...

[AndersHoglund]

Agree, very tricky and too messy to try and hide Huawei problems. Leave it as-is, as it is no big disturbance in HA. OK, will check out the latest version on PR #142 and try it out. But please note, only the kiosk api.

[AndersHoglund]

Updated energy_sensor.py , no immediate issues noted. Will let you know after midnight what it looks like.

[AndersHoglund]

Tested with kiosk and debug log, looks OK. No glitch in lifetime energy data and no other issues in the log.

Funny thing is that Monthly and Yearly data also looks clean now, and I did double check that I have the latest version that only corrects lifetime data. Fix for Yearly and Monthly are not there. Have Huawei cleaned up their act or was there a a problem with the old version? Python magic.... Whatever, all looks well and fine now. Thanks for your effort.

Maybe the lifetime glitch is also gone, and no need for any fix at all? I can not see any glitch in the log. Will test removing that part of the code, stay tuned.... //A

[AndersHoglund]

Interesting and confusing, glitches are back on all three when commenting out the filter that is supposed to catch only LifeTimeEnergy glitch. I will revert that code change and see what Monthly Energy looks like 1:st of Feb.

[tijsverkoyen]

That is really weird. As the Total Life Time Energy will only be updated when the system is producing. Why did you comment that?

[AndersHoglund]

Weird is an understatement. I commented it out to see if maybe Hauwei has fixed something. But no, all glitches came back when commenting out the glitch-filtering. Now restored the code and all glitches are gone on all three entities, Liftime, Yearly and monthly. Do not understand how that is possible. I am waiting for tomorrow, Feb 1st, to see if Monthly energy starts from zero or not. If OK, you can release this update. That is if OpenAPI also works of course.

I am starting thinking this is NOT a Huawei problem, more like an HA coordinator or Python coding issue. Seems as the values for first hour after midnight are picked up from the same hour of the previous day, not the current new day. The bottom value of the glitches have the exact same value as the value at that time of the previous day. Weirdness delux. I am no Python guy to make any sense of this. do not understand any of this indeterministic behaviours we have seen lately, datatypes, glitches etc.

[AndersHoglund]

Today all looked as expected, glitches on Yearly and Monthly energy entities. Monthly data was later reset to zero correctly and cumulativeEnergy was filtered correctly. PR OK for release.

I can not explain what happened after midnight 01-28 where all data looked OK, no glitches. Except maybe for timing and race conditions against Huawei Kiosk server. I have made excerpts of the interesting events in the logs for both 240128 (no glitches) and this night (with glitches), see attached files. Full logs are available on request. /A

2024-01-28 000434.900 DEBUG log No Glitch.txt 2024-02-01 003732951 DEBUG Glitch.txt

[tijsverkoyen]

@AndersHoglund just to be sure: this was with the if statement only for Lifetime?

[AndersHoglund]

Yes, I never uploaded the version with filter for all three entities. I realised it would have other problems, so I never did. I posted a comment on that instead. And as I said earlier, I double checked that I did upload the current latest version with only the lifeTime filter. My experiments with commenting/un-commenting this filter was based on that upload, as per the screenshot above.

[AndersHoglund]

It's all coincidence, an intermittent behaviour from Huawaei I assume. No glitches tonight without any changes on my side. Never mind the gap in the graphs, we had a major fibre outage.

[tijsverkoyen]

I will not try to fix the glitches for Current Month or Current Year. Huawei FusionSolar should up their game and don't send wrong values.

I have created https://github.com/tijsverkoyen/HomeAssistant-FusionSolar/releases/tag/v3.0.3 which has the better handling for the glitches in Lifetime.

[AndersHoglund]

OK, great. Good job. Let Huawei clean up their own dirt. /A