Authentication broken again

[pulsebreaker]

Hi,

Something in the authentication is broken (again) it seems. I'm getting this since 12:05 this afternoon:

code=400, reason=, body=b'{"error":"invalid_grant"}' Traceback (most recent call last): File "C:/Program Files/JetBrains/PyCharm 2023.3.2/plugins/python/helpers/pydev/pydevd.py", line 1534, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2023.3.2\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "F:\Programmeren\PyCharm\netatmo\netatmo_weatherstation-latest.py", line 17, in ws = lnetatmo.WeatherStationData(authorization) File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 282, in init self.getAuthToken = authData.accessToken File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 181, in accessToken if self.expiration < time.time() : self.renew_token() File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 192, in renew_token if self.refreshToken != resp['refresh_token']: TypeError: 'NoneType' object is not subscriptable

I checked my app in Netatmo Connect, it's still there, nothing changed.

Kind regards,

Arno

[rvk01]

I refreshed the Refresh Token and changed the permission on the credentials file. The file was then updated on each code start with a new Refresh Token. Then it worked once with a new Refresh Token, and now it always fails, ant the credentials file does not update.

The library could handle the grant error somewhat better (instead of crashing).

Did you create the refresh-token via the web interface of dev.netatmo.com? Did you also check if the client_id is exactly correct? If it's not it'll also crash with grant error.

[buldre]

Yes on dev.netatmo.com. I updated my first comment, so you'll see I made a careful token transfer;-). My client_ID and client_secret have not changed at all since I first made OAuth work with lnetatmo.

[rvk01]

My client_ID and client_secret has not changed at all since I first made OAuth work with lnetatmo.

No, but netatmo made a change a few weeks ago to test for correct client-id. While you could use a wrong one in the past, you can't anymore and the client-id need to match the one on the dev page exactly. The secret however shouldn't be needed. Do you specify the secret and id somewhere in the code? If you do, that might override something.

[buldre]

No, they are only in the credentials file and are exact copies from the web page

[catalincolesnicov]

I get also following error using the api:

code=400, reason=, body=b'{"error":"invalid_request","error_description":"No \\"refresh_token\\" parameter found"}'
Traceback (most recent call last):
  File "/usr/local/bin/netatmo_influx.py", line 9, in <module>
    weatherData = lnetatmo.WeatherStationData(authorization)
  File "/usr/local/lib/python3.9/dist-packages/lnetatmo.py", line 413, in __init__
    self.getAuthToken = authData.accessToken
  File "/usr/local/lib/python3.9/dist-packages/lnetatmo.py", line 240, in accessToken
    if self.expiration < time.time() : self.renew_token()
  File "/usr/local/lib/python3.9/dist-packages/lnetatmo.py", line 251, in renew_token
    if self.refreshToken != resp['refresh_token']:
TypeError: 'NoneType' object is not subscriptable

[rvk01]

I noted however, when I clicked to copy the tokens on my Netatmo page in Safari and pasted them into a Notes document on my Mac that some capital letters pasted as small caps, so beware! Better to read the Tokens off the Netatmo page and take care to check capital O versus zeros and small caps L versus capital I.

It's strange that some have this problem and others not (although I'm not sure if there are still others with problems)

BTW. The refresh-token and client-id do not contain O or L characters. It's base64 so only contains 0 to 9 and a to f (with a pipe | in the middle for the refresh). All lowercase (although that might not matter). The client-secret does contain other characters. (I now see the secret does get used)

[buldre]

In fact, my client_ID and secret have not changed since I created the App, and that was long before the OAuth. Maybe I should?

[rvk01]

You can always try to create a new app. Also note that "data protection officer" name and email is now mandatory (this also changed recently). Maybe that could be a reason.

[buldre]

What is "data protection officer", and is it used in the library, and do I need to use it in my code?

[rvk01]

No, it's not something you need in your code. But on dev.netatmo.com in your account, it has appeared as an input field (next to the app name etc) which has an asterisks next to it, so it's a mandatory field. You need to fill it in before you can change anything there.

Just put in your own name and email there.

Data protection officers (DPOs) are responsible for monitoring an organisation’s compliance with data protection obligations, informing the organisation of these obligations, and acting as a contact point for data subjects and the relevant supervisory authority. https://www.itgovernance.eu/nl-nl/data-protection-officer-dpo-under-the-gdpr-nl

[buldre]

Thanks! I'll try tomorrow. Bed time in home of yr now :-)

[buldre]

On first start the new app runs. I will wait at least three hours before I stop and try to restart.

Here is what I did: I created a new app which is configured as a copy of my first with a different name. The Client_ID is new and, still 24 characters. The Client_Secret is new and, 44 characters. The old had 37. The first part of the Access Token is the same as the first part of the Refresh Token, 24 characters, and equal to the old. The second part of the Access Token is new and holds 32 characters like the old. The second part of the Refresh Token is new and holds 32 characters like the old.

The credentials file uses TAB to indent each credential. Owner : user Group : user View : Anybody Change : Anybody Execute: Nobody

The second part of the Refresh Token was overwritten on start of the code - which calls the authentication only once. This command runs every 10 minutes: weatherData = lnetatmo.WeatherStationData(authorization) The code imports lnetatmo version 4.1.0. Python 3.11 on Raspberry Pi 4

Crossing fingers!!!

[buldre]

It seems to work with short on/off intervals now, as expected. :-)

[ultima-originem]

My code stopped working on 2024-06-04 with the dreaded "invalid_grant" error. I haven't been successful in solving this problem. Here's what I did:

• updated lnetatmo from 4.0.0 to 4.1.0
• registered a new app, with all new id, secret and token.
• updated lnetatmo.credentials with the new info
• ran my code from a 5m crontab

and here's what happened:

• the authorization immediately started working again and the code ran for the first time at 22:40
• the code stopped running at 02:00 with the same "invalid_grant"error

Can anyone tell me what to do to resolve this issue? I've read the complete thread but am running out of ideas.

[buldre]

Does it matter which name you use for the credentials file? I have not seen it starting with an L before.

Did you check the file permissions for the credentials file? Mine is now running with: View : Anybody Change : Only owner Execute: Nobody

[ultima-originem]

Thanks for pointing that out; this was only a typo in my posting. Here are the real filenames and permissions:

pi@dev[~] ll .net*
-rwxrwxrwx 1 pi pi 189 Jun 23 02:00 .netatmo.credentials*
-rw-rw-r-- 1 pi pi 175 Jun  4 07:00 .netatmo.credentials.2024-06-22

The first one is the current one. It originally had the same permissions as the the second one which was in use for years. I changed the permissions after lnetatmo stopped running at 02:00 (as reported).

Note the timestamp on the original file: 07:00 which was the exact time when the code stopped authenticating.

I also changed the permissions to the ones you're using (-rw-r--r-- 1 pi pi 189 Jun 23 02:00 .netatmo.credentials) and ran the standard test to ensure that my code is not responsible:

pi@dev[~] /home/pi/miniconda3/bin/python3 /home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py
lnetatmo - ERROR: code=400, reason=, body=b'{"error":"invalid_grant"}'
Traceback (most recent call last):
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 1097, in <module>
    weatherStation = WeatherStationData(authorization)                                # Test DEVICELIST
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 416, in __init__
    self.getAuthToken = authData.accessToken
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 241, in accessToken
    if self.expiration < time.time() : self.renew_token()
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 252, in renew_token
    if self.refreshToken != resp['refresh_token']:
TypeError: 'NoneType' object is not subscriptable

Any more ideas/tests?

[rvk01]

The first one is the current one. It originally had the same permissions as the the second one which was in use for years. I changed the permissions after lnetatmo stopped running at 02:00 (as reported).

Note the timestamp on the original file: 07:00 which was the exact time when the code stopped authenticating.

If the script could write the file at Jun 23 at 2:00, it should have no problems reading and overwriting it later. So it can't be a file permission problem.

Either the wrong refresh_token / client_id / client_secret is written to that file... or... netatmo is messing with their systems.

You say you already made a new app (with id and secret etc). Then the data protection officer is also set correctly (I was hoping that that could be the problem if missing).

BTW. You say it stopped working on Jun 23 2:00. Did you mean exactly that time? Because... when the invalid grant happened on 2:00, it shouldn't be writing the credentials to the file (because there aren't any). If the file was written at 2:00 it could crash 3 hours later (when needing a new refresh-token) OR at 2:05 when you run a cron-script again and the refresh-token is used again. But not at 2:00.

You say you started fresh at 22:40 with a new refresh token. If you use cron every 5 minutes, then the refresh-token (and every new refresh-token which is received every 5 minutes) did the job correctly for more than 3 hours. So there MUST be something at the end of netatmo where it goes wrong.

Can you build in a sleep of 10 or 30 seconds in your cronjob. I have this: */5 * * * * sleep 10; /home/pi/netatmo_influx.py > /dev/null 2>&1

Maybe there is a critical second problem at netatmo where it fails when it's run exactly on a certain time (in combination with that 3 hours where the original refresh token would expire).

(I was also thinking about a possible penalty for not reusing access_token each time consistently, but that would mean everyone should have the problem and here it still runs fine.)

[ultima-originem]

You raise some interesting points which I will try to address.

You say you already made a new app (with id and secret etc). Then the data protection officer is also set correctly (I was hoping that that could be the problem if missing).

I did enter the data protection officer and the app did originally run as expected and when it did, it ran between the following timestamps:

FIRST TIMESTAMP: 2024-06-22T20:38:54.000Z
LAST TIMESTAMP:  2024-06-22T23:50:09.000Z

which represents a period of 11475s or 3 hours, 11 minutes, and 15 seconds. The timestamp on the credentials file is 9m51s after the last datapoint.

I have changed my crontab to:

*/5 * * * * sleep 13; /home/pi/scripts/netatmo.py

This fails with the same "invalid_grant" error. I believe I've give you all the relevant data; let me know if I missed something.

[rvk01]

I have changed my crontab to:

*/5 * * * * sleep 13; /home/pi/scripts/netatmo.py

This fails with the same "invalid_grant" error. I believe I've give you all the relevant data; let me know if I missed something.

Did you create a new refresh_token for that and put it in the credential file? After that... did it fail directly?

What scopes did you use to create the refresh_token on dev.netatmo.com?

[ultima-originem]

I did not create a new refresh token today; do you want me to do that? The scope I used was "read_station".

[philippelt]

@ultima-originem there is a strange definition of scopes. You may read https://github.com/philippelt/netatmo-api-python/issues/56#issuecomment-1638546613

[rvk01]

I did not create a new refresh token today; do you want me to do that? The scope I used was "read_station".

Yes, you definitely need to create a new one. The old refres_token had expired and won't work anymore. Once a refresh_token stops working, it'll never work again.

I used read_station and read_thermostat, but I think read_station should be ok.

[rvk01]

@ultima-originem there is a strange definition of scopes. You may read #56 (comment)

Isn't that only if you use homedata (which the default WeatherStationData doesn't use). If you use homedata class, then yes, add those scopes.

[ultima-originem]

I have created a new token, using read_station and that results in:

pi@dev[~] /home/pi/miniconda3/bin/python3 /home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py
lnetatmo - WARNING: Your current token scope do not allow access to Home data
Traceback (most recent call last):
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 1104, in <module>
    homes = HomeData(authorization)
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 605, in __init__
    self.rawData = resp['body']
TypeError: 'NoneType' object is not subscriptable

This is the first time I'm getting the "Your current token scope do not allow access to Home data" warning. Will now retry with read_station and read_thermostat as @rvk01 did.

Result: no change; I'm still getting the same warning.

Based on the comment by @philippelt I generated a new token with the read_presence scope added. Result:

pi@dev[~] /home/pi/miniconda3/bin/python3 /home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py
lnetatmo - WARNING: No Smokedetectors found
lnetatmo - WARNING: No Cameras found
lnetatmo - WARNING: No Persons found
lnetatmo - WARNING: No events found
lnetatmo - WARNING: No thermostat avaible for testing
lnetatmo - WARNING: Your current token scope do not allow access to HomeCoach
Traceback (most recent call last):
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 1124, in <module>
    Homecoach = HomeCoach(authorization)
  File "/home/pi/miniconda3/lib/python3.9/site-packages/lnetatmo.py", line 955, in __init__
    self.rawData = resp['body']['devices']
TypeError: 'NoneType' object is not subscriptable

The response with the new token and scope is fundamentally different from the responses before today, when I always got "invalid_grant".

Although I get the warnings shown above when I run lnetatmo.py, I noticed approx. 15m later, that my code is now authenticating correctly (with the 3 scopes mentioned earlier). I plan to leave this alone for at least 3hrs. to get past the time where authentication may stop again and report back.

[rvk01]

@ultima-originem Yes, you didn't mention you used HomeCoach(). You had WeatherStationData in your error there. For HomeCoach you need a more extensive scope. The normal WeatherStationData just needs read_station.

Hope it keeps working past 3 hour for you now.

[ultima-originem]

@rvk01 This is a misunderstanding which probably originated from the post by @philippelt who asked me to look at this comment

Based this comment I expanded the scope from read_station with read_presence and later also with read_thermostat and this is the scope which I'm currently using. This expanded scope generated the warning when running lnetatmo.py but is also delivering data.

What in your opinion should the scope be for reading the weather station and rain data? I can try a reduced scope once I'm past the 3hr. period.

[rvk01]

What in your opinion should the scope be for reading the weather station and rain data? I can try a reduced scope once I'm past the 3hr. period.

I'm using read_station and read_thermostat successfully with just WeatherStationData, with which I read temperature, co2 (inside) and wind and rain (outside modules). But I don't think read_thermostat is needed. Just read_station.

BTW. I see you called lnetatmo.py library directly. I think that might crash on HomeData() if you don't have that. Don't you have your own script which just calls WeatherStationData() ?

[ultima-originem]

I tried your scope earlier today and at that time it failed. When I added readpresence it worked but I can't escape the feeling that Netatmo's responses aren't consistent or always predictable. I run my own script and that's working now (for weather station and rain) but also called lnetatmo directly because that's the recommended test in the docs (from the docs: If you provide all the values in a credential file, you can test that everything is working properly by simply running the package as a standalone program.)_. I don't recall any mentioning of HomeData() in that context.

[rvk01]

I tried your scope earlier today and at that time it failed. When I added read_presence it worked but I can't escape the feeling that Netatmo's responses aren't consistent or always predictable.

Don't use the lnetatmo.py script directly. It has a demo mode where it also calls that homecoach (when called directly) which will fail because of limited scopy. Only use an own script with just a call to WeatherStationData. (You can see what I mean when you look at the bottom part of lnetatmo.py)

[ultima-originem]

I agree but would like to note that apparently the docs are misleading/incomplete in this case. Thanks for clearing that up.

[rvk01]

Yes. The library could handle the errors returned better (instead of erroring out with that NoneType error). Also note it errored out on HomeCoach so the already called WeatherStationData was successful. But it does show that the library can't handle unexpected results.

[ultima-originem]

It's 18:02 here and Netatmo is still authenticating which is very nice but I stil don't know why now and not earlier.

[philippelt]

There is obviously an intermittent error on Netatmo side. Others have reach the same conclusion : https://helpcenter.netatmo.com/hc/en-us/community/posts/19320250276626-API-invalid-grant-at-random-intervals?page=2#comments

[ultima-originem]

My code is still authenticating almost a day after it started. For now I'm going to leave my scope as is, as experience has shown that it may be difficult to get it running again. The root cause is still unknown. Thanks to @philippelt, @rvk01 and @buldre for responding to my post.

[LarsJensen69]

This error must be at Netatmo. I have 2 RPI connected to 2 different Netatmo stations. One in Denmark and one in Spain. When this issue occur it is on both of them at the same time +/-2 minutes which is my checking interval. They gave absolut nothing in common apart from both authenticating at Netatmo. I made a Try-Except-Finally so it reboots automatically every time it fails.

[LarsJensen69]

Any chance of getting an update according to the changes described by Netatmo? https://helpcenter.netatmo.com/hc/en-us/community/posts/20156931057170/comments/20162263407634

[rvk01]

Any chance of getting an update according to the changes described by Netatmo? https://helpcenter.netatmo.com/hc/en-us/community/posts/20156931057170/comments/20162263407634

@LarsJensen69 This has already been implemented in the latest version. The refresh_token is renewed every time and is stored in the credential file.

[philippelt]

The initial issue raised can now considered be closed so I am going to lock and close this topic to avoid confusion.