Issue with User-Agent field from #36 is happening again

[muppet3000]

36 is happening again, we need to think of a longer term solution, however a short term fix will be to change the User-Agent field again.

@indykoning - I'll email you directly about this over the weekend to discuss possible alternative implementations. (I'd like to keep the chat on implementation private so that we can reach a good consensus on it).

[Sjord]

Could it be that Growatt is purposely blocking other clients? Perhaps someone's that using this libary is querying too often, and Growatt responded by blocking the user-agent.

It would be nice to know what's going on on Growatt's side of things. One possibility is to identify properly in the user-agent header, e.g. by including the URL to this repo. That would risk getting blocked easily again, but also informs Growatt of this alternative client and hopefully get in contact.

[muppet3000]

@Sjord - Yes that's exactly what they're doing, it's most likely because of the home-assistant integration polling the API every 1 minute for people's installations. I've had some ideas about how I plan to implement this - part of which will involve attempting to speak to Growatt (but I don't hold out much hope).

In the meantime, anyone using this library directly can simply override the headers value by doing something like this:

headers = {"User-Agent": "Dalvik/2.1.0 (Linux; U; Android Device)"}
api.session.headers.update(headers)

You can just set the value of User-Agent to whatever you want, so long as it's not what the library is currently using.

[skunktrading]

Surely the correct fix is limiting the frequency of the polling that "home-assistant integration" is doing. It seems pointless to call more often than once every 300 seconds anyway.

[muppet3000]

Yes a drop in the frequency will definitely be part of the fix, however I disagree with 300 seconds as the interval, my system for example provides updates to the growatt servers every minute and that means that data changes as often as every 2 minutes. But even an increase to every 2 minutes would halve the number of requests hitting the API.

[skunktrading]

My growatt inverter only updates every 5 minutes. Sorry, I erroneously assumed they were all like this.

[indykoning]

Yeah the HA integrations update every 5 minutes which i suggest not doing it any faster than that. The default settings of the wifi sticks are every 5 minutes so you don't gain any useful data. And it prevents spamming those servers any more than you need.

I agree with Sjord that it is good etiquette to inform them about the package used to access it so they could get in contact. If it is User Agent based we could get around it by using random id's in the User Agent, but i'd prefer not being blocked in the first place.

[muppet3000]

I'm pushing an update to the HA integration now, it will bump the library version and I've also increased the polling to 5 minutes as suggested but I plan to submit a better fix in the future to allow people to choose this value from a drop-down. For now I just wanted to get a release out that 'fixes' the current problem. I have some time tomorrow night that I'm going to spend putting together options for a proper fix as well as contacting growatt about the throttling.

[muppet3000]

I've submitted a PR which I think goes a long way to addressing this issue. It even means that some of us can use a custom User-Agent to provide information to Growatt in the hope they reach out and engage over proper support of the library. Happy to receive feedback on it.

[indykoning]

This should be fixed with the new PRs implemented in https://github.com/indykoning/PyPi_GrowattServer/releases/tag/1.3.0 For now i'll close this, if it happens again we'll reopen this or create a new issue 🙂

[muppet3000]

@indykoning - The latest built-in agent_identifier (https://github.com/indykoning/PyPi_GrowattServer/blob/master/growattServer/__init__.py#L28) has already been blacklisted by Growatt, even if you add a random string to the end of it with the add_random_user_id option. However, the ability to provide your own agent_identifier string means people can work around this by generating their own randomly or dynamically (that's how I'm doing it in the Home Assistant integration).

If you want to change/update the default agent_identifier then great but given there's a suitable option available in the library I'd say to just leave it now.