500 response requires component reload

[dbarrosop]

Hello, since the upgrade I am experiencing an issue where the request to zaptec's API starts failing after a certain amount of time (this has happened twice after >1 day of operation since I upgraded on Friday) and requires reloading the component. This is the error in the server logs:

This error originated from a custom integration.

Logger: custom_components.zaptec
Source: custom_components/zaptec/__init__.py:207
Integration: Zaptec EV charger (documentation)
First occurred: 03:53:02 (412 occurrences)
Last logged: 07:22:02

Fetching data failed: RequestError: GET request to https://api.zaptec.com/api/installation/<id> failed with status 500: <ClientResponse(https://api.zaptec.com/api/installation/<id>) [500 Internal Server Error]> <CIMultiDictProxy('Date': 'Mon, 20 Nov 2023 06:17:44 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Vary': 'Origin', 'Request-Context': 'appId=cid-v1:<id>', 'X-Rate-Limit-Limit': '30m', 'X-Rate-Limit-Remaining': '29998', 'X-Rate-Limit-Reset': '2023-11-20T06:30:00.0000000Z', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains')>
Fetching data failed: RequestError: GET request to https://api.zaptec.com/api/circuits/<id> failed with status 500: <ClientResponse(https://api.zaptec.com/api/circuits/<id>) [500 Internal Server Error]> <CIMultiDictProxy('Date': 'Mon, 20 Nov 2023 06:18:24 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Vary': 'Origin', 'Request-Context': 'appId=cid-v1:<id>', 'X-Rate-Limit-Limit': '30m', 'X-Rate-Limit-Remaining': '29999', 'X-Rate-Limit-Reset': '2023-11-20T06:30:00.0000000Z', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains')>
Fetching data failed: RequestError: GET request to https://api.zaptec.com/api/chargers/<id>/settings failed with status 500: <ClientResponse(https://api.zaptec.com/api/chargers/<id>/settings) [500 Internal Server Error]> <CIMultiDictProxy('Date': 'Mon, 20 Nov 2023 06:20:45 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Vary': 'Origin', 'Request-Context': 'appId=cid-v1:<id>', 'X-Rate-Limit-Limit': '30m', 'X-Rate-Limit-Remaining': '29999', 'X-Rate-Limit-Reset': '2023-11-20T06:30:00.0000000Z', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains')>
Fetching data failed: RequestError: GET request to https://api.zaptec.com/api/installation/<id> failed with status 500: <ClientResponse(https://api.zaptec.com/api/installation/<id>) [500 Internal Server Error]> <CIMultiDictProxy('Date': 'Mon, 20 Nov 2023 06:21:15 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Vary': 'Origin', 'Request-Context': 'appId=cid-v1:<id>', 'X-Rate-Limit-Limit': '30m', 'X-Rate-Limit-Remaining': '29990', 'X-Rate-Limit-Reset': '2023-11-20T06:30:00.0000000Z', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains')>
Fetching data failed: RequestError: GET request to https://api.zaptec.com/api/chargers/<id>/state failed with status 500: <ClientResponse(https://api.zaptec.com/api/chargers/<id>/state) [500 Internal Server Error]> <CIMultiDictProxy('Date': 'Mon, 20 Nov 2023 06:22:02 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Vary': 'Origin', 'Request-Context': 'appId=cid-v1:<id>', 'X-Rate-Limit-Limit': '30m', 'X-Rate-Limit-Remaining': '898', 'X-Rate-Limit-Reset': '2023-11-20T06:30:00.0000000Z', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains')>
Traceback (most recent call last):
  File "/config/custom_components/zaptec/__init__.py", line 207, in _async_update_data
    await self.account.update_states()
  File "/config/custom_components/zaptec/api.py", line 1007, in update_states
    await data.state()
  File "/config/custom_components/zaptec/api.py", line 231, in state
    data = await self.installation_info()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zaptec/api.py", line 377, in installation_info
    data = await self._account._request(f"installation/{self.id}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zaptec/api.py", line 932, in _request
    raise log_exc(
custom_components.zaptec.api.RequestError: GET request to https://api.zaptec.com/api/installation/<id> failed with status 500: <ClientResponse(https://api.zaptec.com/api/installation/<id>) [500 Internal Server Error]>
<CIMultiDictProxy('Date': 'Mon, 20 Nov 2023 02:53:02 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Request-Context': 'appId=cid-v1:<id>', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains')>

Maybe an expired access token? This problem never manifested prior to the upgrade.

As mentioned above, reloading the component fixes the problem.

[sveinse]

Let's see. There is a special status code (401) for refreshing access tokens, so the response of error code 500 is an odd response. My next idea would be that the access rate limit has been exceeded. However, the zaptec docs clearly states that the error code 429 Too many requests are used for that. Evidently the zaptec cloud is doing something else.

The reason this starts with this version, is that the previous integration were rather permissive on data fetching. This version logs any error. Would you say in your opinion that it's better to silently ignore this type of error (where the sensors aren't updated) than logging it?

update Reloading the component fixes it? This is interesting and must be investigated. Do yo have opportunity to run the special zaptec-dev version if I make a special version that can be instrumented?

[dbarrosop]

There is a special status code (401) for refreshing access tokens, so the response of error code 500

Might not be related, I just thought it could've been that because it happens after a certain amount of time and once it starts happening not a single request works until the component is reloaded.

429 Too many requests

Yeah, don't think it's that either the log shows the following headers are present in the response:

'X-Rate-Limit-Limit': '30m', 'X-Rate-Limit-Remaining': '898'

Would you say in your opinion that it's better to silently ignore this type of error (where the sensors aren't updated) than logging it?

The problem is that when this happens all sensors show as "Unavailable" and no action succeeds. They all fail with this 500 error. If this was a transient issue that happens on some requests I guess it'd be fine to ignore them but this renders the component unusable.

Do yo have opportunity to run the special zaptec-dev version if I make a special version that can be instrumented?

That should be fine. Just let me know the branch/commit sha and I will update it.

[sveinse]

I just pushed a new version in master that I'm curious will fix the issues you are observing. Can you please install this and try if that goes haywire after some time.

[RuneOynes]

Hi. Just installed zaptec integration. when i em trying to add Integration, i get an error message: Feil Konfigurasjonsflyt kunne ikke lastes inn: 500 Internal Server Error Server got itself in trouble

Is this same problem? Can i help with some logfiles? :-)

[sveinse]

@RuneOynes which version of the zaptec integration are you running? You should be running the "master" version, as it contains some recent fixes

[dbarrosop]

@sveinse will do but to be honest I haven't seen that issue again since I reported it so feel free to ignore it until for now.

[sveinse]

I think it is important to mention that Zaptec had (or maybe still have) issues with the cloud stability. Since the HA integration is completely dependent on the cloud, this instability will reflect itself to the zaptec integration as well.

That said, I am interested in making the zaptec integration as robust as possible for these kinds of problems. So I encourage having discussions like this.

[RuneOynes]

I em running 0.0.5

[sveinse]

@RuneOynes, ok. Please install the latest beta 0.7.0 (or even master), as the 0.0.5 has been replaced.

[RuneOynes]

I installed/upgraded to master 0.7..XXX, adding integration showed the setup page, and everything seems to work ok.
By the way, i had zaptec in ECO mode. And with eco mode i was not able to start or stop charging. No error message. Thanks

[sveinse]

@RuneOynes please add a separate issue for that