Netatmo integration: API error: 500 (Internal Server Error)

rosch100 commented 3 months ago

The problem

Since the upgrade to 2024.6.1, the Netatmo integration stopped working. The error is:

Logger: homeassistant.components.netatmo
Quelle: components/netatmo/__init__.py:90
Integration: Netatmo (Dokumentation, Probleme)
Erstmals aufgetreten: 10:48:25 (7 Vorkommnisse)
Zuletzt protokolliert: 10:51:14

API error: 500 (Internal Server Error)

What version of Home Assistant Core has the issue?

Version core-2024.6.1 Installationstyp Home Assistant OS Entwicklung false Supervisor true Docker true Benutzer Virtuelle Umgebung false Python-Version 3.12.2 Betriebssystemfamilie Linux Betriebssystem-Version 6.6.28-haos-raspi CPU-Architektur aarch64 Zeitzone Europe/Berlin Konfigurationsverzeichnis /config

What was the last working version of Home Assistant Core?

2024.6.0

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Netatmo

Link to integration documentation on our website

https://www.home-assistant.io/integrations/netatmo/

Diagnostics information

home-assistant_netatmo_2024-06-09T08-54-46.855Z.log

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

I could (temporarily) resolve the issue by deleting and re-adding the integration. However, today the integration fails again with the same error. Deleting and Re-adding the integration is not a solution!

domibarton commented 2 months ago

Of course, if only I'd have a clue what the issue would be. I cannot reproduce and the integration is utilising the authentication mechanism of HA.

I checked the Netatmo HA integration for a few minutes, but couldn't figure out how the whole oauth thingy works, resp. saw that it's using HA's internals to do it. Can we check the HTTP 500 response, resp. the payload of it? I'm a Python developer, but didn't do much with HA :/

Of course, if only I'd have a clue what the issue would be.

Isn’t it about this? https://helpcenter.netatmo.com/hc/en-us/community/posts/19320250276626/comments/19329138660370

I already mentioned that above, seems it can be related…

If that would be the case wouldn't everyone have to have the same issue?

tbh, I didn't had the problem until I restarted HA. Other people seem to have the same issue (after restarting). Some said it works after re-adding the integration, some claimed it failed again after a few hours or days.

domibarton commented 2 months ago

Sorry, should've read the logger docs. I'll try to configure the logger for the Netatmo integration and see if I can get more details – at least more than HTTP 500 (which actually by RFC definition is a server, and not a client error… but I don't trust it in this case ^^).

domibarton commented 2 months ago

So, this is going to be hard to debug, because:

Activating debug for the Netatmo integration didn't help
The HTTP 500 came from NabuCasa, and not Netatmo directly

Debugging the error with those 2 limitations (esp. the last one) is quite hard. I suspect (or hope?) the Netatmo API will give a more useful error, but NabuCasa just throws away that «origin & hopefully useful error message» and shows us a HTTP 500 instead (similar to this issue – not the cause, but the error handling) 😄. This is bad error handling practice par excellence (from the HA core / NabuCasa cloud, not the Netatmo integration), but there might be a reason for the obscurity :/

At least when there's an error on the NabaCasa integration, then a HTTP 500 might be «correct» in that context. Still annoying to debug. Or does anyone know if I can access «my» NabuCasa logs?

cgtobi commented 2 months ago

But, is it really related to authenticating via NabuCasa? I understood that also people with individual Netatmo dev accounts had that issue. I just tried both setups on my dev environment without issues which is not helping either. Nor has my production env (dev account) any issues of that sort. So It is really difficult to find a relation here.

domibarton commented 2 months ago

But, is it really related to authenticating via NabuCasa? I understood that also people with individual Netatmo dev accounts had that issue. I just tried both setups on my dev environment without issues which is not helping either. Nor has my production env (dev account) any issues of that sort. So It is really difficult to find a relation here.

I agree, it's a strange case, and quite hard to debug, or even to find any relations. I think it's related to the token renewal, at least that's the endpoint which is throwing an HTTP 500:

Failed setup, will retry: 500, message='Internal Server Error', url=URL('https://account-link.nabucasa.com/refresh_token/netatmo')

Of course it can be a side-effect of something else, but right now everything is pointing in that direction, and we don't have anything else to start with :/

Since I'm using HA core with venv's, I can easily clone my live venv to my test venv, then remove & re-add the Netatmo integration to see if it works (again). However, if there's something fucked up with the tokens (or other side effects) on NabuCasa, this can probably «solve the issue once and for all». Which means, the chances I can't reproduce the error in the live environment again are quite high. It's a bit like Pandora's box :/

However, at least we then know it's a NabuCasa (or cloud or whatever) issue and not a local one. If I can reproduce the error after that in live again, then it might be a local one.

I'd love to keep my broken integration in place, so I can reproduce the error and can help out – esp. because you're currently trapped in «works-on-my-machine trap» and you can't do anything about it :)

Do you have any tips on how to debug this issue, for example logging HTTP payloads of requests & responses of my local HA, as well as the ones NabuCasa is sending?

domibarton commented 2 months ago

I understood that also people with individual Netatmo dev accounts had that issue.

Q: Is the oauth flow, resp. the «token handling», implemented the same way in both cases?

As mentioned above, Netatmo did change their oauth implementation around that time, to match the official specs. Which means, they changed how tokens are invalidated. It's a pretty wild guess, but probably its an unfortunate «timing issue» or «racing condition» for some users, which probably led to existing tokens to be invalidated (i.e. invalid_grant), and /authorize won't be executed yet / again.

I know, it's just a wild guess. But guessing is the only option without seeing the origin error or alike :/

wolph commented 2 months ago

Since the error is at the nabucasa domain it's obviously something that's wrong on the home assistant (nabucasa) end. Netatmo might have changed something that broke it, but an internal server error should never happen :)

So I'm not sure who has the logs for the nabucasa side of things but it would be good to know what's happening over there.

FoxScout commented 2 months ago

As I unterstand from last messages, The required update token was not given in time, so will still failing while is not update first? ¿Is there a way from the console in HA to push this refresh_token process? I don't know if there is a line to put in the console one time to push manually this refresh_token and the Netatmo integration works again?

wolph commented 2 months ago

I've personally just removed the integration and added it again. If you keep the entity names the same you will keep the history

FoxScout commented 2 months ago

well.. after a crash updating to 2024.7.3 from 2024.7.1... I reinstall NETATMO, deleting before that the 3th party permisions. And the names changed before that, the attribute's names were like .temperaturE (french) and now are .temperaturA (spanish).

I've lost all records, I think. I search by the french names, but HA doesn't purpose any coincidences.

wolph commented 2 months ago

well.. after a crash updating to 2024.7.3 from 2024.7.1... I reinstall NETATMO, deleting before that the 3th party permisions. And the names changed before that, the attribute's names were like .temperaturE (french) and now are .temperaturA (spanish).

I've lost all records, I think. I search by the french names, but HA doesn't purpose any coincidences.

If you can track down the exact entity ids you were using before you'll have your history back. Hass doesn't delete history at all from what I've seen.

I used the dashboards i was previously using to find the original names.

nifoc commented 1 month ago

Maybe I can add a slightly new perspective: I connected Netatmo to my HA installation for the first time yesterday and ran into this error today (very roughly 24 hours after adding the integration for the first time).

Since account-link.nabucasa.com throws an Internal Server Error shouldn't there be something in the server logs to help debug this specific issue?