Growatt integration missing but basic entities

[BigFam194]

The problem

If adding an optional smart meter to the Growatt solar installation you feed additional data from the smart meter to the Growatt inverter. This extra information is exposed in Growatt app and the Growatt portal but is not visible as new additional entities in the integration. In addition to the basic solar production data and the inverter itself you also get (net) energy consumption of your house, export and import from grid data, these data entities are not surfaced by the integration. After installing the smart meter, the Shine Phone app automatically displays the new additional information but no additional entities show up in home assistant. Tried to restart ha and also to remove and add the Growatt integration with same results.

What version of Home Assistant Core has the issue?

2022.11

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Growatt

Link to integration documentation on our website

https://www.home-assistant.io/integrations/growatt_server/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @indykoning, @muppet3000, @jasperplant, mind taking a look at this issue as it has been labeled with an integration (growatt_server) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `growatt_server` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Change the title of the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign growatt_server` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

growatt_server documentation growatt_server source _{^{(message by IssueLinks)}}

[bartpaeleman]

Sounds exactly the same as the problem I have reported with the sensors that I lost after the upgrade where I had all of my battery information coming from. @BigFam194 do you happen to talk about a datalogger like this one that I too got added when my batteries were installed?

[BigFam194]

Bart, yes it's a smart meter (data logger) of the same make but another model. The entities that were there before are still there but no additional coming from the smart meter.

[muppet3000]

Hi Both,

Thanks for logging this, I'm guessing what has happened is that when Growatt were making all the changes last week they changed something in the API that means your batteries that used to be presented through one method in the API are not longer presented in the same way. Due to the way that this integration works we're unable to see what the people at Growatt are planning and therefore have to react. Ultimately, if these values are still available in the ShinePhone app, it means we can get them back it just may take a bit of effort.

If you're willing to share your credentials with me (my contact info is on my Github page) then I can try and log into your systems myself and work out what additional API calls need to be made to get these sensors back. I can't promise it'll be quick though as this is something I only work on in my spare time and there are lots of other issues that have been tagged against this integration that I've been promising to work on for some time.

I will update this ticket later with a link to a forum post that I'm making with a prioritised list of all of the tickets.

In the meantime @bartpaeleman - You posted some really useful 'before' and 'after' screenshots on this ticket: #80950 earlier, would you mind copy/pasting them over to this ticket so I can track it all in one place?

Edit: Can you please also tell me what type of system you have i.e. the bit I've circled below from the Growatt Web UI:

[muppet3000]

Hi All,

This is a massive cross-post on all the Growatt tickets so apologies for it's generic format. I've collated all of the open issues and prioritised them here: https://community.home-assistant.io/t/growatt-integration-prioritised-list-of-features-for-implementation-fixing/483850

I will be working through them in order and updating that post as well as the affected tickets. Using that method I can be sure to correctly track everything associated with the integration.

Watch this space!

[Saentist]

There is no Second and Third Phase values in three phase inverters

Inverter have two string connectors integration show 3 strings ;) Growatt app show 8 ;) ;)

[bartpaeleman]

Upon request from @muppet3000 posting the information before/after the new release got installed here, with no batteries shown in the end. I lost the following sensors which as you can see in this screenshot did work before:

• sensor.thuis_batterij_lifetime_total_energy_input_1
• sensor.thuis_batterij_lifetime_total_energy_input_2
• sensor.thuis_batterij_lifetime_energy_output

There are a lot more sensors but the above were used on the energy dashboard:

BEFORE THE UPGRADE ENERGY VIEW:

AFTER THE UPGRADE ENERGY VIEW:

I do think that my data comes from a datalogger device that was installed when I had (Growatt) batteries added to my configuration but it was working before so no clue what I can do to get that back?

[bartpaeleman]

Can you please also tell me what type of system you have i.e. the bit I've circled below from the Growatt Web UI:

This is what I see:

UPDATE: @muppet3000, temporary credentials sent to you for access to my system.

[bartpaeleman]

@muppet3000 did you receive my email?

[charlie063]

same issue here, after last HA update, no more data... if, here you can find the API documentation

https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

[muppet3000]

@muppet3000 did you receive my email?

What email? I've only seen notifications in this thread from you?

[muppet3000]

same issue here, after last HA update, no more data... if, here you can find the API documentation

https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

No more data just for the batteries?

[charlie063]

same issue here, after last HA update, no more data... if, here you can find the API documentation https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

No more data just for the batteries?

no data at all... where I have two inverters, on single phase, on three phase, if you can share me who does have the batteries and BMS on stock? -european supplier preferred-

[Saentist]

same issue here, after last HA update, no more data... if, here you can find the API documentation

https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

Outdated API from 2016-12-20

@charlie063 GWL - Elerix

[charlie063]

same issue here, after last HA update, no more data... if, here you can find the API documentation https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

Outdated API from 2016-12-20

@charlie063 GWL - Elerix

do you have a new API manual?

[muppet3000]

Not sure what you're asking about stock for, but when I get around to looking at this issue (which is nearing the top of my list) I'll reach out for credentials from you guys so I can take a look and work out why things have stopped working for you.

Please keep a track of things on this forum post: https://community.home-assistant.io/t/growatt-integration-prioritised-list-of-features-for-implementation-fixing/483850 where I'm updating what I'm working on.

[muppet3000]

same issue here, after last HA update, no more data... if, here you can find the API documentation https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

Outdated API from 2016-12-20 @charlie063 GWL - Elerix

do you have a new API manual?

There is no new API manual, we reverse engineer it from the Android Application, there is no official support from Growatt via an API

[charlie063]

same issue here, after last HA update, no more data... if, here you can find the API documentation https://growatt.pl/wp-content/uploads/2020/01/Growatt-Server-API-Guide.pdf

Outdated API from 2016-12-20

@charlie063 GWL - Elerix

got it working... mamma, I made the default error... rather username I used to registered email address. now I threw myself a fishbone

[bartpaeleman]

@muppet3000 did you receive my email?

What email? I've only seen notifications in this thread from you?

email with credentials to log into my system

[muppet3000]

@muppet3000 did you receive my email?

What email? I've only seen notifications in this thread from you?

email with credentials to log into my system

Got it this time, responded to you directly.

[muppet3000]

Hi Everyone, this is now the next item on my list, it's going to take me a little while to get my head around though.

What would be REALLY helpful is if someone can send me a screengrab from the shinephone app and highlight the specific values on there that are now missing from home-assistant.

What I think has happened is that when they (growatt) made all of the changes the other week they changed where this information came from.

@bartpaeleman - Thanks for the screengrab above with all of the missing values and what they used to be shown as, that's REALLY helpful, it shows me that it was previously identifying with Growatt as a 'tlx' system, but something may have changed and I need to re-map some values.

Could someone please send me a screengrab of all of the entities that are working and all that aren't please, then I can work out what's missing.

[Saentist]

@muppet3000 I. no string connected II. Inverter don't have third string

missing: L2 and L3 https://github.com/home-assistant/core/issues/81470#issuecomment-1302713005

UE series Inverter /6000UE/

it will be good to see Mode and Status of Inverter and Signal and Status of Logger

[bartpaeleman]

@muppet3000 I had all my entities on a single test page too:

[muppet3000]

Thanks @Saentist - Can you also please send me a screengrab from the ShinePhone app on your mobile phone and circle the values on there so I can match them up?

NOTE - My time is going to be focussed on a a proper fix to #81951 currently, but as soon as I've got on top of that I'll be back looking at this again!

[Saentist]

[muppet3000]

Thanks - just what I needed, I think they must have just changed the api call. Can you tell me the names from the screengrabs above that relate to the values that are now missing in the Home Assistant integration? That way when I next look at this I can map the values and work out why they're missing.

[Saentist]

Vac[R,S,T] (V) Correspond to L1 L2 L3 L2 and L3 missing In ShinePhone cannot see some of element with can be seen in web portal. Will upload when there is Sun above horizon.

[danant96]

Hi Both,

Thanks for logging this, I'm guessing what has happened is that when Growatt were making all the changes last week they changed something in the API that means your batteries that used to be presented through one method in the API are not longer presented in the same way. Due to the way that this integration works we're unable to see what the people at Growatt are planning and therefore have to react. Ultimately, if these values are still available in the ShinePhone app, it means we can get them back it just may take a bit of effort.

If you're willing to share your credentials with me (my contact info is on my Github page) then I can try and log into your systems myself and work out what additional API calls need to be made to get these sensors back. I can't promise it'll be quick though as this is something I only work on in my spare time and there are lots of other issues that have been tagged against this integration that I've been promising to work on for some time.

I will update this ticket later with a link to a forum post that I'm making with a prioritised list of all of the tickets.

In the meantime @bartpaeleman - You posted some really useful 'before' and 'after' screenshots on this ticket: #80950 earlier, would you mind copy/pasting them over to this ticket so I can track it all in one place?

Edit: Can you please also tell me what type of system you have i.e. the bit I've circled below from the Growatt Web UI:

In relation to the first post, I too have an Eastron smart meter and in the web interface, the values for import, export, load consumption are displayed in the Single Exportlimit device type. Integration shows values from the inverter but not from the datalogger associated with the Single Exportlimit. Could that be the issue with some of the missing values?

[muppet3000]

Hi Guys, I've just sat down to try and reproduce this and I've re-read all of the messages from above, I've also setup a dev version of HA to look at @bartpaeleman 's system using the credentials he provided. I cannot work out what is missing based on what has been sent above. This is a screengrab of all of the entities I was provided when I logged in with Bart's credentials:

You can get the same page by clicking on this button on the integrations page:

@bartpaeleman - Please can you give me the names or a screengrab of the entities that are missing so that I can try and find them.

There are only 21 possible entities supported for the 'tlx' type in the integration and all 21 of them are showing on my page, the additional 6 are the 'total' values that are shown for all systems, these also appear to be showing correctly.

[bartpaeleman]

@muppet3000 I am getting the same at the moment but before had 3 devices (3rd one was where my battery entities wer part):

These are still remaining from that as ghosted entities in my home assistant:

[bartpaeleman]

I also used them on my energy configuration"

[muppet3000]

Ahhh, I think I understand. So you had the following devices:

1. Totals (for the whole system)
2. zonne_energie (which is the Solar PV panels and the inverter)
3. batterij (which is specific to the battery)

It's the entire batterij (battery) device that is no longer there - If you're able to confirm that it would be very helpful.

I understand better now. I'll try and setup a few more tests over the weekend, they must have changed the way that the battery information is stored and represented on the API.

[bartpaeleman]

Correct! Only the first 2 are available now.

[muppet3000]

Hi again,

Of the values that were previously available what were the ones that you actually used in Home Assistant? It looks like the api calls have changed a bit and I need to re-factor the code to make sure the correct values are provided for the battery. I'm also unsure how it will perform for people that do not have a battery. I'm guessing the values you've provided in the screengrab above e.g. lifetime energy output is one of the key ones (So you can determine what the battery is outputting to the house etc)? Are there any other key ones? I've been busy tonight submitting the PR for the library bump, but this one is then next on my list (I've already had the app working on my phone pointed at your system @bartpaeleman so I already have a few ideas on how to resolve it. I'll just need to put some examples together.

One thing that would be REALLY helpful is if you're able to send me a screengrab from the ShinePhone app and highlight/circle the specific values that you want to be able to get him home assistant, that way it makes it much easier for me to map them onto the API calls.

[HilbertusO]

Hey there . Thanks for all the effort. I don't have a battery, but did make use of (the most basic) 'total lifetime energy output' values (for the 'plant' and per converter, as I own 2). Your recent fix worked perfectly for some days, after which they apparantly changed the API yet again.. (?)

2 screendumps to pointing at those values (the shineapp converter-screen is so crappy, the value is actually outside of the view :))

Just let me know if you need anything else.

[bartpaeleman]

Hi @muppet3000 thanks for all your efforts and fantastic work.

This is what I get for the battery view (which in my case comes under Storage named "Zonne-energie and not the "Meter" choice if that means something to you at all.

It then gives this view when I open it:

When I then look in my ShineApp under the "all parameters" link, there are a ton of values that mean nothing to me immediately... Let me know if you need a screenshot of all. However using some rational logic, the name of these and the fact that they have some real values, may appear to me as potentially useful for that purpose. But it could also be that EpvTotal )sum of the Epv1 and Epv2 is the one that was used or the eChargeTotal (kWH) and eDischaredTotal(kWh) values. I would think that all the values with E in front are potentially useful.

[charlie063]

@muppet3000 here we go, please look that I share two inverters on 1 plant. this to other side of the winddirection of the farms, where I think ppv1 and ppv2 for the 8000 model are key. where pv1 and pv2 are for the MIC xxxx TLX model?

there is no soc connected yet.

I hope this info might bring you further in your adventure trying to make the world a better place. my support you have ;)

[muppet3000]

values (the shineapp converter-screen is so

The most recent fix for the User Agent problem is now fixed as of this morning, you should be able to take the latest nightly build tomorrow (1st December) and it should start working again.

[muppet3000]

Thanks @charlie063 - So you're saying you have no battery system connected to your system? That's actually really helpful because in your screenshots there are clearly parameters listed for where the battery values would be. Therefore, when I add the functionality to make the battery work again for @bartpaeleman it shouldn't break for you. It would be really helpful though to test against your system as well before I push all the changes.

[muppet3000]

@bartpaeleman - Thanks for your screengrabs. It looks like the 'all parameters' button isn't available on the Android version of the App!!!! I think that's what we used originally to implement this feature though.

I think I can now work out what's needed to re-add the battery though, but it will appear under the main entity, rather than it's own system.

What I was actually after is specifically what values you used to make use of in Home Assistant when it worked. The ones you're missing are anything 'useful' from the battery right?

If I was able to get the amount charged & discharged from the battery both daily & lifetime would that solve your missing values? I think I can probably also get the SOC (percentage) value as well. I'm going to play around some more and see what I can get.

The confusing/worrying thing is that the Android app isn't making the same API calls that it used to so I have no idea how long the API call will still exist. But given that the iPhone app is still supporting it then I think they've actually just forgotten to include it on the Android version!

[muppet3000]

Ok, update from tonight's investigations.......

I have worked out how to get almost all of the missing battery values back into the system, in fact I think I can improve on what was there before. It won't appear in the same way that it did before but that's no problem really, if anything it's an improvement. (If you're interested in seeing what values will be re-added look at the code block below)

On top of that, I've found out that the TLX systems share almost all of the same API calls (just with different names) as the Mix system. This means I can massively improve what's on offer for these systems and implement it almost identically to Mix (which is what my own system is).

I'm going to implement this in two phases.

1. Changes just to Home Assistant to add in the values that are already available with the data that we query
2. Make changes to the upstream library to support the additional API calls correctly, then make HA call them and bring in all the extra functionality

This is just a brain dump from myself so I don't lose my progress (if you're interested though, these are the extra values already available that I just need to add in to the HA Integration to make the battery appear again):

#Solar Production
epv1Today: PV1 Energy Today kwh
epv1Total: PV1 Energy Total kwh
Up to 4
epvTotal: Total of all epv kwh

#Discharging
edischargeToday: Battery discharged today kwh
edischargeTotal: Battery discharged total kwh
bdc1DischargePower: Battery 1 discharging power w
bdc1DischargeTotal: Battery 1 discharged total kwh
Up to 2

#Export to Grid
etoGridToday: Exported to Grid today
etoGridTotal: Exported to Grid total

#Load consumption
elocalLoadToday: Load consumption today kwh
elocalLoadTotal: Load consumption total kwh

#Charging
echargeToday: Battery charged today kwh
echargeTotal: Battery charged total kwh
bdc1ChargePower: Battery 1 charging power w
bdc1ChargeTotal: Battery 1 charged total kwh
Up to 2

#Others
bmsSoc: SoC

[charlie063]

Ok, update from tonight's investigations.......

I have worked out how to get almost all of the missing battery values back into the system, in fact I think I can improve on what was there before. It won't appear in the same way that it did before but that's no problem really, if anything it's an improvement. (If you're interested in seeing what values will be re-added look at the code block below)

On top of that, I've found out that the TLX systems share almost all of the same API calls (just with different names) as the Mix system. This means I can massively improve what's on offer for these systems and implement it almost identically to Mix (which is what my own system is).

I'm going to implement this in two phases.

1. Changes just to Home Assistant to add in the values that are already available with the data that we query
2. Make changes to the upstream library to support the additional API calls correctly, then make HA call them and bring in all the extra functionality

This is just a brain dump from myself so I don't lose my progress (if you're interested though, these are the extra values already available that I just need to add in to the HA Integration to make the battery appear again):

#Solar Production
epv1Today: PV1 Energy Today kwh
epv1Total: PV1 Energy Total kwh
Up to 4
epvTotal: Total of all epv kwh

#Discharging
edischargeToday: Battery discharged today kwh
edischargeTotal: Battery discharged total kwh
bdc1DischargePower: Battery 1 discharging power w
bdc1DischargeTotal: Battery 1 discharged total kwh
Up to 2

#Export to Grid
etoGridToday: Exported to Grid today
etoGridTotal: Exported to Grid total

#Load consumption
elocalLoadToday: Load consumption today kwh
elocalLoadTotal: Load consumption total kwh

#Charging
echargeToday: Battery charged today kwh
echargeTotal: Battery charged total kwh
bdc1ChargePower: Battery 1 charging power w
bdc1ChargeTotal: Battery 1 charged total kwh
Up to 2

#Others
bmsSoc: SoC

@muppet3000 why total per day? and not polling by per hour? now? I get one piek in the graph rather spread over the wol graphs data?

[muppet3000]

Because that's what's available on the API, HomeAssistant does some really clever logic to monitor values for 'increasing totals' and then calculates the hourly rates from them, trust me, this definitely works. The energy dashboard works off of increasing totals not current values, I can assure you this is correct. It works exactly the same for all other systems and the energy dashboard works correctly (with the exception of one that has a slight bug in it on the API side, but that's an anomaly not the general rule). I should be able to put together a proof-of-concept tomorrow night and then leave it testing over the weekend and share screen grabs etc. I don't have much time over the weekend though so it's tomorrow night or I'll have to do it next week instead.

[charlie063]

Thanks @charlie063 - So you're saying you have no battery system connected to your system? That's actually really helpful because in your screenshots there are clearly parameters listed for where the battery values would be. Therefore, when I add the functionality to make the battery work again for @bartpaeleman it shouldn't break for you. It would be really helpful though to test against your system as well before I push all the changes.

@muppet3000 correct, no SOC connected, on purpose, since I am working with another doc manufactory to develop something everybody is looking for (I hope)

[charlie063]

Because that's what's available on the API, HomeAssistant does some really clever logic to monitor values for 'increasing totals' and then calculates the hourly rates from them, trust me, this definitely works. The energy dashboard works off of increasing totals not current values, I can assure you this is correct. It works exactly the same for all other systems and the energy dashboard works correctly (with the exception of one that has a slight bug in it on the API side, but that's an anomaly not the general rule). I should be able to put together a proof-of-concept tomorrow night and then leave it testing over the weekend and share screen grabs etc. I don't have much time over the weekend though so it's tomorrow night or I'll have to do it next week instead.

@muppet3000 you've got mail :)

[charlie063]

@muppet3000 look at this one, from today :D

[muppet3000]

I think that's maybe the bug I was talking about yesterday, there's another ticket for that. I'll dig it out later. I should be able to replicate it myself when I have the Proof of Conecpt I talked about yesterday up and running

[muppet3000]

We have lift-off:

This system is now showing all of the extra values associated with the battery as well as a load of extra ones I found as well.

There is still no way to determine the energy drawn from the grid at this point, that will require a new API endpoint on the upstream library.

I've set the Energy Dashboard up like this:

I'm going to leave it running over the weekend and see what all the metrics look like.

@bartpaeleman - A huge thank-you for letting me have access to your system to implement this. There's still a load more work I need to get done, but this is a step in the right direction!

[bartpaeleman]

@muppet3000 Hey Chris, this is awesome!

At first glance this looks perfect and even better than what I had since before I only had one entity available (defined by battery lifetime output) to add to my energy configuration screen and had to repeat it for both battery in and out values:

I noticed in the new setup that you have a ton more useful values such as both the total charged and discharged, which I think is fantastic! As for your access, I am going to keep that in place for the benefit of the community and yourself, for as long as you need it.

As for the energy drawn, I am personally getting that from my digital meter P1 port and not from my Growatt system anyway.

Again, thanks a ton!!!

[muppet3000]

Hi all, just an update on this, I've been running this since last week now and things seem to be providing the correct value. There are a few caveats though.

1. It doesn't have all the values that I want it to have - but this requires additions to the upstream library to make it work
2. It suffers the same problem that has been identified in other tickets (that I will dig out and link here later) where the values drop to zero overnight.

Therefore, I would like to continue looking at this until at a minimum I have fixed item (2) above. However, given the time of the year and the fact I have other personal commitments I may merge the changes so far so that people can get some of the functionality and then add in the fixes/additions for (1) & (2) separately when I next have time to work on it properly.

I'll update again later in the week.