Discrepancies between app and integration totals

[lozzd]

Version of the custom_component

0.0.20 (via HACS) - according to manifest.json, const.py lists 0.0.18

Configuration

Using API key, 15 second polling I have a Zappi and Harvi currently (PV generation, grid, car charging)

Describe the bug

I've been happily using this integration to pull my myEnergi numbers in to HA for a couple of months - thanks! - but I just noticed today that the numbers in the myEnergi app/website don't match those in HA.

As of 13:45 today, my numbers are:

myEnergi app Generated: 13.7 kWh Imported: 4.1 kWh Exported: 4.4 kWh Home Consumption: 8.7 kWh Zappi: 4.7 kWh

Home Assistant: sensor.myenergi_my_house_generated_today: 9.46 kWh sensor.myenergi_my_house_green_energy_today: 3.19 kWh(??) sensor.myenergi_my_house_grid_export_today: 3.1 kWh sensor.myenergi_my_house_grid_import_today: 3.16 kWh

I tried to start adding and subtracting numbers to see if I could make sense of this but I really can't...

(My smart meter's import measurement so far today is 3.65kW, which doesn't really tally with either of those..)

While it's worrying they're all slightly wrong, I'm mostly concerned that the PV generation is massively off. I've just realised that maybe my panels were far more efficient than I thought.

Note, this only seems to affect the totals - the live numbers seem to be spot on.

Any idea what's going on here?

[lozzd]

The pattern of the graph looks pretty normal by the way, so it's just like there's some multiplier on the actual value that's distorting the numbers - this is "generated today" - the app shows the total ended yesterday at 13.1kWh, versus less than 10kWh from the integration.

Since the "today" numbers are used in the HA Energy dashboard, this does affect price calculation etc of grid usage.

[G6EJD]

I’ve just checked mine, the alignment between systems is good: SolarEdge Generation: 11.06kWh MyEnergi Home Generated Today: 11.52kWh Shelly Import: 11.44kWh GivEnergy: 10.93kWh

So all near perfect alignment given that a CT has an accuracy of +/-3%

Is your system Single Phase? Are you sure the Grid CT is on the incoming Grid feed and not a spur?

[lozzd]

Yes, the CT is correctly located. I'm more worried that the data from the same source (myEnergi) is totally different between their app and this integration. Maybe it's a question for the underlying python library?

[G6EJD]

When Johan created the integration I ran the Python query for a whole day and got the same result, so I think it’s ok.

You can run the script from the HA command line, the results are in Joules-sec so for a day the result needs to be divided by 60*60 to convert to Wh then 1000 for kWh.

Your Smart meter is the reference and your readings are at 30-50% higher, so something strange is going on.

Most MyEnergi users get good correlation to about 3%, so I think that confirms their system is good.

I s just checked mine again: SolarEdge 13.1kWh MyEnergi 13.1kWh

How does the MyEnergi App totals compare with the Integration?

[lozzd]

Sorry to clarify the numbers in my first post are myEnergi app versus the integration... so totally different. Disregard the single smart meter reading for now...

[G6EJD]

OK, so do you think your python scripts are up-to-date?

[lozzd]

I believe so. The const.py here does stil say 0.0.18 but it's running 0.0.20.

When Johan created the integration I ran the Python query for a whole day and got the same result, so I think it’s ok.

Can you elaborate how to do that so I can verify on the CLI?

As of 16:39, the myenergi app reports generation of 17.1kWh and the integration reports 12.25kWh!

[G6EJD]

See here: https://github.com/cjne/pymyenergi

noting the ratio is not constant between the two readings, it’s getting close to 1:1

[lozzd]

Thanks!

bash-5.1# myenergi --password <redacted> --username <redacted> overview | grep generated
Energy generated : 13.55kWh

Unsurprisingly, that matches what HA says. but the myenergi app says 17.7kWh!

So I assume this issue is for cjne/pymyenergi then?

[lozzd]

I'll have to take a meter reading on my generation meter tonight and compare to tomorrow night to see what's actually true here. Now I'm very confused..

[G6EJD]

Do you see a full response from 00:00, the API returns from 00:00 to the current time.

The question is, how come mine is perfectly aligned between GA, App and SolarEdge results?

[lozzd]

How can you tell? Certainly the HA graphs all increment from midnight (UK time) so I should expect so.

I was starting to think maybe an app bug, but if yours align... hmm.

Just checked the clock on the zappi too just in case... all good.

[lozzd]

Results of today (21st, mostly cloudy) Certified generation meter: 12.83kWh Myenergi app: 12.9kWh (rounds to one decimal place) Home Assistant/pymyenergi: 9.65kWh

Well. at least the CTs are working as expected.

That leaves us looking at pymyenergi and wondering why it's so different to the official app....

[G6EJD]

I definitely think your Python code is out of date, my results: SolarEdge 14.0 kWh MyEnergi App: 14.1 kWh HA: 14.1 kWh Shelly: 14.0 kWh

Try removing the Integration, then delete the Python folder and reinstall.

[lozzd]

I'll do that tomorrow. I've just noticed the value in HA is now 10.02kWh... so increased 0.37kWh despite the "now" (obviously) being flat for hours since the sun went down...

[lozzd]

Removed, HACS deleted the directory, restarted HA, re-installed repo from HACS, re-added the integration - showing 1.18kWh in HA/python, app is showing 2kWh.

So there is still a difference between pymyenergi and the app

[G6EJD]

All I can observe is there are probably 000’s using this integration and if there was an endemic fault then it would have emerged by now and others would be reporting the same.

The question is what is different with your environment?

I’m using the entity: myenergi_yourname_generated_today My is actually myenergi_home_generated_today

Have you removed old entities from HA it tends to retain everything, my system has 43 entities and 4 devices for Myenergi

I’ve just checked my 4 systems (SolarEdge, myenergi App and HA and a Shelly monitor) they are all giving identical readings.

[lozzd]

All I can observe is there are probably 000’s using this integration and if there was an endemic fault then it would have emerged by now and others would be reporting the same.

Yes, I tend to agree and if I could figure out what was happening I wouldn't have opened this! I do wonder if it's not affecting more people but they haven't noticed because the numbers seem very feasible - they're off 10-20% after all.

The biggest giveaway should have been the fact the HA number continues to increment after the sun set, but I hadn't noticed that until now.

I haven't bothered touching the entities as we now know this is not a HA problem - pymyenergi shows it without HA's involvement (yes the generated number from that output matches exactly the entity myenergi_<home>_generated_today)

So given both pymyenergi and the Myenergi app use the same "API", I'm struggling to figure out what's different here.

A snapshot of today so far:

bash-5.1# myenergi --password <redacted> --username <redacted> zappi energy
generated: 3.03kWh

App:

HA:

[lozzd]

Can you do a debug run -d on your end to see if it matches this? It's a shame debug mode doesn't output raw data, I might put a PR in to add that so we can inspect it...

bash-5.1# myenergi -d --password <redacted> --username <redacted> zappi energy
DEBUG:pymyenergi.client:Refreshing data for all myenergi devices
DEBUG:pymyenergi.connection:Get Myenergi base url from director
DEBUG:httpx._client:HTTP Request: GET https://director.myenergi.net/cgi-jstatus-E "HTTP/1.1 401 Unauthorized"
DEBUG:httpx._client:HTTP Request: GET https://director.myenergi.net/cgi-jstatus-E "HTTP/1.1 200 OK"
INFO:pymyenergi.connection:Updated myenergi active server to https://s18.myenergi.net
DEBUG:pymyenergi.connection:GET /cgi-get-app-key- https://s18.myenergi.net/cgi-get-app-key-
DEBUG:httpx._client:HTTP Request: GET https://s18.myenergi.net/cgi-get-app-key- "HTTP/1.1 401 Unauthorized"
DEBUG:httpx._client:HTTP Request: GET https://s18.myenergi.net/cgi-get-app-key- "HTTP/1.1 200 OK"
DEBUG:pymyenergi.connection:GET status 200
DEBUG:pymyenergi.connection:GET /cgi-jstatus-* https://s18.myenergi.net/cgi-jstatus-*
DEBUG:httpx._client:HTTP Request: GET https://s18.myenergi.net/cgi-jstatus-* "HTTP/1.1 401 Unauthorized"
DEBUG:httpx._client:HTTP Request: GET https://s18.myenergi.net/cgi-jstatus-* "HTTP/1.1 200 OK"
DEBUG:pymyenergi.connection:GET status 200
DEBUG:pymyenergi.client:Adding zappi zappi-<redacted>
DEBUG:pymyenergi.client:Adding harvi harvi-<redacted>
DEBUG:pymyenergi.base_device:Fetching hour history data for zappi
DEBUG:pymyenergi.connection:GET /cgi-jdayhour-Z<zappi SN>-2022-8-23-0-24 https://s18.myenergi.net/cgi-jdayhour-Z<zappi SN>-2022-8-23-0-24
DEBUG:httpx._client:HTTP Request: GET https://s18.myenergi.net/cgi-jdayhour-Z<zappi SN>-2022-8-23-0-24 "HTTP/1.1 401 Unauthorized"
DEBUG:httpx._client:HTTP Request: GET https://s18.myenergi.net/cgi-jdayhour-Z<zappi SN>-2022-8-23-0-24 "HTTP/1.1 200 OK"
DEBUG:pymyenergi.connection:GET status 200
generated: 2.22kWh

[lozzd]

I finally got around to play some more. Does this output look sane?

[{'dom': 25, 'dow': 'Thu', 'gen': 3240, 'imp': 2340840, 'mon': 8, 'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gen': 3180,
  'hr': 1,
  'imp': 2189400,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gen': 2700,
  'hr': 2,
  'imp': 2206620,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'exp': 900,
  'gen': 2520,
  'hr': 3,
  'imp': 1835100,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gen': 2700,
  'hr': 4,
  'imp': 1881660,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gen': 2460,
  'hr': 5,
  'imp': 1637040,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gen': 3120,
  'gep': 1080,
  'hr': 6,
  'imp': 1529700,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gen': 2160,
  'gep': 113760,
  'hr': 7,
  'imp': 2882760,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gep': 405780,
  'hr': 8,
  'imp': 1629600,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'gep': 717540,
  'hr': 9,
  'imp': 1552860,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'exp': 12960,
  'gep': 1543740,
  'hr': 10,
  'imp': 1620900,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'exp': 1135080,
  'gen': 1260,
  'gep': 3363960,
  'hr': 11,
  'imp': 144900,
  'mon': 8,
  'yr': 2022},
 {'dom': 25,
  'dow': 'Thu',
  'exp': 34560,
  'gep': 589080,
  'hr': 12,
  'imp': 938640,
  'mon': 8,
  'yr': 2022}]

I assume this is in UTC which is why I only have entries up to hr 12, is the first row that misses hr 0? so 12-1am?

I did also check and there's no difference between if I switch between the history_energy_minutes and history_energy_hours calls.

[lozzd]

I'm losing my mind here. I just checked "imported" on the myenergi website for yesterday and it reports 24.31kW (not even close) but their app reports 15.3kWh! Their own website and app dont' agree let alone HA (11.43kWh)

I think I'm going to go to myenergi support at this stage......

[sddgit]

I installed this integration just a couple of hours ago, and am seeing the same problem:

HA integration Generated Today: 38.53 Myenergi app Generation: 47.9 HA Energy component Solar Production: 46.5 (driven by Enphase Envoy feed) Enphase web site 46.6

HA integration Imported Today: 15.53 Myenergi app Imported: 17.9 Myenergi web site: 17.43 HA Energy component Imported: 17.2 Enphase web site 16.8

I also see the great disparity between the app and web site for yesterday: Myenergi app imported yesterday: 50.8 Myenergi web site imported yesterday: 68.3 HA Energy component Imported: 50.1 Enphase web site 49.8

Do the “today” sensors come directly from the API, or are there calculations involved? I wonder if the definitions of today, yesterday etc are wrong because of some time zone problem between the API, app and web site.

[G6EJD]

All the data comes from the MyEnergi servers via the API.

I note on the Myenergi forum that some customers report inaccuracies during the day but these get corrected the next day, so if you check yesterday against a known reading there should be parity. The problem is therefore in the day.

All data is recorded in UTC.

I don’t see any difference in readings, suggesting the code here is OK.

[lozzd]

I'm glad I'm not on my own here @sddgit. I've not heard anything back from my support case with myenergi. I'll let you know when I do.

[sddgit]

@G6EJD Regardless of any intra-day discrepancies that might exist, the fact that, for example, today’s generation is wildly different between the integration and the app would indicate something’s amiss somewhere. They are using, presumably, the same API, so why would that be the case? Do you know if the “today” value comes directly from the API, or is it a calculation that is performed? Perhaps it comes back as some sort of value that must be corrected based on time zone? Clutching as straws here! As I said, “yesterday” values are different between the app and the web site, so that’s another problem, but outside the scope of the integration.

@lozzd What time zone are you in, out of interest?

[lozzd]

I'm in the UK so currently BST (UTC+1). Reading the python code everything is set to use UTC, which makes sense. i don't think it's timezone related because I can see the "generation today" increment at, say, 10pm when I've been at 0 watts generated (due to the sun being down...) for many hours, so 1 hour offset is enough to disprove that (I think)

Btw if you want to see the API examples they're in my output above. If you get CLI access to your HA instance you can run the myenergi command to see the numbers (which exactly match what's in HA). If you run with debug mode, you can see the URLs, which make sense to me - they're constructed of the serial of the zappi, the date (today), the hour to start (0, e.g. midnight) and the number of hours (24, makes sense). I've tried understanding the JSON response but I don't get it. The numbers even for 1am-2am show generation. The app doesn't though! So maybe they switched the app to use a different API and some people's original API are broken based on their account? My account is only a couple of months old, I assume yours is also new? (I only got my zappi/hub/harvi in July)

[sddgit]

Yes, my Zappi is only a couple of months old (a V2.1 one). Do those “today” numbers come through as a number that’s been calculated server side, do you know?

[lozzd]

Yes, although there is a bit of calculation to convert it to kWh. See post https://github.com/CJNE/ha-myenergi/issues/213#issuecomment-1227198196 and you'll see the JSON response, it's hourly so the gen numbers are added up and then turned in to kWh.

So the actual totalling of power generated by hour is done on the server side, yes.

[sddgit]

Right. But the integration adds up all the hours and converts to kWh?

[lozzd]

Yes. Or rather technically the pymyenergi library does, and the integration uses that library.

[sddgit]

Understand. Let’s hope you get a response from myenergi regarding this. I suppose it’s feasible there is something going astray in the Python library that doesn’t affect everyone somehow. All very strange.

[lozzd]

Does anyone know the difference between gep and gen in the API response? Sometimes it has one or the other and sometimes both

[yacoob]

I think I'm seeing similar problem here. Much simpler instalation, zappi with internal hub, no solar panels. The charger is in use right now, and a charging session has started roughly a minute past midnight. With 01:20AM on the clock, both myenergi app and myenergi website show that the charge is proceeding at 7.7kW, with 10.4kWh already imported from the grid. sensor.myenergi_hub_grid_import_today in HA shows 4.41kWh. sensor.myenergi_hub_power_import looks sensible, hovering at 8kW, which is correct, entailing 7.7kW for the charger and ~300 kW for the house. sensor.myenergi_zappi_energy_used_today is also at 4.41kWH.

Happy to debug more here, if you let me know what you'd like me to check :)

[lozzd]

Welcome to the club. myenergi has yet to respond to my support ticket - still open. if it goes on much longer I'll ping "Dr Chris", their CTO who is very active on their forum. It sounds like at least a few people have got totally random unreliable numbers across their platform which is concerning.

I was starting to wonder the other day if my installer didn't do something quite right. Maybe others can chime in if they have a similar setup? I have the Zappi, installed on a circuit not via the consumer unit. I then have a Harvi with 2x CT's - one on the incoming mains (before both Zappi and consumer unit, right after meter) and one on the generation. The Zappi is the "master" and configured to read the CT's from the Harvi.

It seems that whenever the Zappi reboots (e.g. I did a firmware update the other day, but just a reboot will do) the Harvi comes unpaired with the Hub (light goes red - and if I reboot the Hub, it goes out completely!). During the time the Harvi light is red or off, the app still shows the live power stats just fine - I guess the data flow is going Harvi -> Zappi -> Hub - but the API we're using stops updating anything under the "Harvi" section (timestamp halts too), so no live metering is done at all at all. This has led me to believe that the app is using a totally different API than the cgi- endpoint used by the myenergi python code. I have to then re-pair the Harvi with the hub, and when I do the numbers start updating in the API again and thus in Home Assistant. Very odd.

I haven't even broched the subject of why there's a (much much smaller) discrepancy between my smart meter IHD and the myenergi app. Honestly I don't trust the Smart Meter at all - it's one of the Landis & Gyr SMETS1 models that's only just been "re-activated" to be smart, and the IHD only updates the live readings every 2-3 minutes. I can turn the kettle on, watch the myenergi app instantly show it, and the IHD for the smart meter can be blissfully unaware still showing 300 watts and the kettle finishes before it updates... Useless!

[G6EJD]

The Harvi only sends data to the Master which sends data via the Hub, the Master is the hub not the Hub. My data is still in prefect alignment as is most users. The Smart meter records all usage and updates the display and providers servers periodically, but they are accurate.

[yacoob]

Two questions:

• for the users that do see discrepancies; do we have a confirmation whether it's the data coming out of myenergi API (that ha-myenergi uses) that is invalid, or is that data correct and it gets miscounted within the lib or HA integration code?
• I wonder what is the hardware that people in this thread are using; maybe it's related to the v2 of zappi having an integrated hub?

[lozzd]

I will try and figure out item 1 today. It's not a complicated piece of maths from the look of the code, so I am pretty sure it's the API endpoint. As I say, I don't think the app is using that endpoint any more. Need to find someone who wants to decompile the Android app and poke around maybe.

I have a Zappi that was manufactured earlier this year, it's a v2 (as indicated by the firmware it runs) but I don't think it has a hub. I have a separate hub (and the harvi)

[DReffects]

Hey Guys, I've just installed Home Assisant and this awesome integration today, so everything is shiny and new.

I am noticing the very same deveations:

For today:

App: Generated total: 24.7 kWh Green energy used: 11.8 kWh Green energy exported: 12.7 kWh Imported energy: 9.1 kWh

Total energy used: 20.7 kWh Car charged: 2.72kWh Usage of our home:: 18.3 kWh

These numbers also add up with the readings at myaccount.myenergi.com

Home Assistant integration reports:

sensor.myenergi_xxx_generated_today: 20.63 kWh (still rising despite being night atm) sensor.myenergi_xxx_green_energy_today: 2.3 kWh (is this the amount of green energy to my car? I have set a value of 80% green energy with eco+ mode) sensor.myenergi_xxx_3_grid_export_today 10,79kWh (still rising despite being dark at the moment) sensor.myenergi_xxx_3_grid_import_today: 7.73 kWh

Any idea what is going on here?

[lozzd]

That sounds like the exact problem yep. It's the generation raising past sunset that's the giveaway. It does seem like they fixed the website to match the app. I haven't heard back from support, despite also PMing Dr Chris. Getting a bit frustrating now. I've got an Eddi coming next week and I'm somewhat worrying if I got in to the wrong ecosystem

[DReffects]

I am rather new to the whole myenergi system, Dr Chris was very helpful in the past since my installer sold me a pre-used hub that had another zappi linked.

I've opened up a topic on their forums: https://myenergi.info/reported-api-values-are-extremely-different-from-a-t6655.html

Perhaps you could chime in there.

I absolutely love the "true solar surplus charging" with the zappi.

[lozzd]

I found it quite interesting that my stats updated in HA totally fine the entire time the app was down this morning.

Dr Chris replied to my support ticket and suggested that it could be data loss, since there is some "data infill" that happens on the app to mitigate loss of transmission. However, I pointed out that I have no yellow (!) and asked for clarification on whether that happens server side or client side - since if it's on server side the numbers should still match, right?

[G6EJD]

How does their system know to flag a yellow if no data was ever sent, are they counting samples expected out of 3600 or 60 or whatever and if the answer doesn’t match flag data not received? Who knows. I have never had a different result between API and App and I’m sure it’s a poor radio link performance for those that have the issue. My radio link is rock solid.

[lozzd]

How does their system know to flag a yellow if no data was ever sent, are they counting samples expected out of 3600 or 60 or whatever and if the answer doesn’t match flag data not received?

Yes.

My radio link is rock solid.

My equipment is 2 meters apart with no obstructions. So pretty sure mine is too.

I pointed this out and haven't had a response in a few days. Will update when I hear anything back

[sddgit]

How does their system know to flag a yellow if no data was ever sent, are they counting samples expected out of 3600 or 60 or whatever and if the answer doesn’t match flag data not received? Who knows. I have never had a different result between API and App and I’m sure it’s a poor radio link performance for those that have the issue. My radio link is rock solid.

I have a Zappi with built-in hub, so no radio link involved, and no warning triangles. And if it were a radio link problem, wouldn’t it affect api, app and web about equally?

[lozzd]

By the way - after my Eddi installation yesterday I will note that my "green energy today" number is totally matching the app. So at least one metric is correct between the app and API/HA!

[lozzd]

First full day of having the Eddi paired - still using the same Hub, Zappi is still the master, Harvi still has both CTs attached........ And now my numbers are correct???

App Gen / HA Gen: 12.8kWh / 12.75kWh App Import / HA import: 8.4kWh / 8.21kWh App Export / HA export: 3.6kWh / 3.58kWh

So... has adding the Eddi fixed an issue in pymyenergi, or has it fixed something on the server? 🤔

[sddgit]

Too much to ask to remove/replace the Eddi to see if anything changes?

[lozzd]

I think that'll be quite hard to do. It just automatically adds to your account as soon as the Hub sees it.

So two things happened on the morning I had the Eddi installed. They also had that long maintenance.

When I looked at my data in the app a few hours later, all of my history for my grid and generation, but not the house had been wiped and started fresh. The forum could not tell me if that was due to the Eddi installation or the maintenance.

If everyone else's is still wrong/out of sync and they have their historical data, then it was the Eddi and not the maintenance...

[sddgit]

I have my history, no Eddi, and my “today” numbers don’t match between the app and HA.

[pannlov]

I have also the exact same problem. I can take my grid import yesterday as an example. MyEnergi app, MyEnergi webportal, my solar power meter and my Energy Power Company web portal all reports about 32 Kwh. myEnergi HA integrations reports 23 kwh. Sp for sure the HA integration is incorrect. It would be a really good integration if it works with correct values, but with that much discrepancies it is of no use. I hope this will ne fixed and keep up the good work!