Authentication broken again

[pulsebreaker]

Hi,

Something in the authentication is broken (again) it seems. I'm getting this since 12:05 this afternoon:

code=400, reason=, body=b'{"error":"invalid_grant"}' Traceback (most recent call last): File "C:/Program Files/JetBrains/PyCharm 2023.3.2/plugins/python/helpers/pydev/pydevd.py", line 1534, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2023.3.2\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "F:\Programmeren\PyCharm\netatmo\netatmo_weatherstation-latest.py", line 17, in ws = lnetatmo.WeatherStationData(authorization) File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 282, in init self.getAuthToken = authData.accessToken File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 181, in accessToken if self.expiration < time.time() : self.renew_token() File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 192, in renew_token if self.refreshToken != resp['refresh_token']: TypeError: 'NoneType' object is not subscriptable

I checked my app in Netatmo Connect, it's still there, nothing changed.

Kind regards,

Arno

[rvk01]

Are you working with the .netatmo.credentials file for authentication? Is the content of that .netatmo.credentials valid? And what time/date does that file have?

(For me it still works.)

[pulsebreaker]

I do use the file and the contents are still the same. I checked my script on two locations (NAS and local, both with the same credentials file) and both produce the same result. Weird. 

It still works for you so I will check my end again, thanks. 

Arno

---

Van: rvk01 @.***> verzonden: maandag 3 juni 2024 18:08 Aan: philippelt/netatmo-api-python Cc: pulsebreaker; Author Onderwerp: Re: [philippelt/netatmo-api-python] Authentication broken again (Issue #82)

Are you working with the .netatmo.credentials file for authentication? Is the content of that .netatmo.credentials valid? And what time/date does that file have? (For me it still works.) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[rvk01]

I just got word from another user (with a raspberry pi reading out netatmo) where the Grafana view also isn't updated from around 12:00 today (last entries 11:45 CEST I see). So there might indeed be a problem. (I need to investigate this after diner ;) ).

Weird that it works fine at my end (with the same account as the other user).

[philippelt]

could you check that you are using the last version of the library (from repo, not yet from pip ) ?

[pulsebreaker]

Okay, no worries. 

I my case it looks like it had something to do with the Refresh Token. I did a refresh on the Netatmo website, copied the new Refresh Token in the credentials file and it started to work again. 

Maybe it had something to do with write-permission on my credentials file? I changed it to 'everyone is allowed to write' to see if this helps in the future. 

Enjoy your diner. 

Arno

---

Van: rvk01 @.***> verzonden: maandag 3 juni 2024 19:12 Aan: philippelt/netatmo-api-python Cc: pulsebreaker; Author Onderwerp: Re: [philippelt/netatmo-api-python] Authentication broken again (Issue #82)

I just got word from another user (with a raspberry pi reading out netatmo) where the Grafana view also isn't updated from around 12:00 today (last entries 11:45 CEST I see). So there might indeed be a problem. (I need to investigate this after diner ;) ). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[pulsebreaker]

Okay, my 'solution' didn't work, it just works 1 time, if I try for a second time I get the same error.

I was using version 3.0.0. Updated it to 4.0.0, same error.

Then I downloaded the repo and did an overwrite of the lnetatmo.py file with the one from the zipfile, same result:

code=400, reason=, body=b'{"error":"invalid_grant"}' Traceback (most recent call last): File "C:/Program Files/JetBrains/PyCharm 2023.3.2/plugins/python/helpers/pydev/pydevd.py", line 1534, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2023.3.2\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "F:\Programmeren\PyCharm\netatmo\netatmo_weatherstation-latest.py", line 17, in ws = lnetatmo.WeatherStationData(authorization) File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 413, in init self.getAuthToken = authData.accessToken File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 240, in accessToken if self.expiration < time.time() : self.renew_token() File "C:\Users\signu\AppData\Local\Programs\Python\Python37-32\lib\site-packages\lnetatmo.py", line 251, in renew_token if self.refreshToken != resp['refresh_token']: TypeError: 'NoneType' object is not subscriptable

If I manually generate a token on the Netatmo site and copy it in the credentials files it works again but just 1 time.

Regards,

Arno

Op ma 3 jun 2024 om 19:18 schreef Philippe Larduinat < @.***>:

could you check that you are using the last version of the library (from repo, not yet from pip ) ?

— Reply to this email directly, view it on GitHub https://github.com/philippelt/netatmo-api-python/issues/82#issuecomment-2145739773, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF5ASQG2LXZAODH4HSWNWN3ZFSQN5AVCNFSM6AAAAABIWZ2Y4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBVG4ZTSNZXGM . You are receiving this because you authored the thread.Message ID: @.***>

[philippelt]

Sorry but I didn't updated the pypi version as I wanted to finish some refactoring.

Could you pick the library from github directly (4.1.0) ?

[rvk01]

Haha. The really weird part.... the installation that I mentioned (where it stopped working around 12:00 CEST) was running my own implementation in PHP. I don't use a webcreated token there but use OAuth2 for that (loging in and allowing access). It fails with invalid_grant directly trying to get a new refresh token.

My own installation here, with this netatmo-api-python code, still works fine, and still retrieves new access_tokens. I can see that from the time for the .netatmo file.

In my own implementation (at the remote site) I get a code via OAuth2 which I need to pass with grant_type' => "authorization_code". There it failes with {"error":"invalid_grant"}. But the code here doesn't use that (I think). So there are multiple issues at the end of netatmo.

(For the installation here I use 4.0.0 and still works.)

[philippelt]

Many thanks @rvk01 for your feedback.

So it looks like Netatmo is playing again with authentication ! My various tasks are still working here but I don't know if it will last long !

Thanks for any news/feedback in this thread...

[pulsebreaker]

I directly downloaded the code from here, still the same issue.

It worked flawlessly for the last year or something until this afternoon so I suspected something changed or got broken at the Netatmo side.

The windmeter broke a couple of months ago also btw. I received no more winddata, only the battery status, time etc. and that was also in the official app. When it drained it's new batteries within a month I gave up on that.

Kinda done with the expensive unreliable stuff from Netatmo.

Regards,

Arno

[rvk01]

It worked flawlessly for the last year or something until this afternoon so I suspected something changed or got broken at the Netatmo side.

I suspect that newly created access_tokens have problems with being passed with (any) grant_type. Older tokens seem to work fine. Maybe there was a problem around 12:00 and newly created tokens have that problem.

Authentication site is also kinda slow. And now I sometimes get a "500 COMMON.SERVER ERROR" on https://api.netatmo.com/oauth2/authorize

Guess we need to wait a bit to see if it resolves itself.

[rvk01]

Aaaaah. I've got my own PHP implementation working again :) At the end it seems I used a client_id for doing OAuth2 which was different from the one for retrieving the tokens.

I'm not sure if it's the same problem in the code in this library (because it doesn't do OAuth2).

I did change another thing but I'm not sure if it could also be an issue. When entering the app page on dev.netatmo.com, I saw that there is a (new) mandatory field "data protection officer name" & email. Those had asterisks (so mandatory) but where not filled in (they were not there when the app was registered). But because I filled them in before I saw the actual problem with the client_id, I'm not sure if it might be an issue.

Hopefully it keeps working here...

[pulsebreaker]

Just tried again, no changes. It did work, it doesn't work now.

Arno

Op ma 3 jun 2024 om 22:26 schreef rvk01 @.***>:

Aaaaah. I've got my own PHP implementation working again :) At the end it seems I used a client_id for doing OAuth2 which was different from the one for retrieving the tokens.

I'm not sure if it's the same problem in the code in this library (because it doesn't do OAuth2).

I did change another thing but I'm not sure if it could also be an issue. When entering the app page on dev.netatmo.com, I saw that there is a (new) mandatory field "data protection officer name" & email. Those had asterisks (so mandatory) but where not filled in (they were not there when the app was registered). But because I filled them in before I saw the actual problem with the client_id, I'm not sure if it might be an issue.

Hopefully it keeps working here...

— Reply to this email directly, view it on GitHub https://github.com/philippelt/netatmo-api-python/issues/82#issuecomment-2146061683, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF5ASQELSNURRCWTGUGAFT3ZFTGQXAVCNFSM6AAAAABIWZ2Y4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGA3DCNRYGM . You are receiving this because you authored the thread.Message ID: @.***>

[rvk01]

Just tried again, no changes. It did work, it doesn't work now.

Did you add the data protection officer name & email in the app registration page?

I just tried it with freshly created token via the web interface and it worked here. (used user root instead of normal pi to test) Or does it work for you for 3 hours and then fails?

[pulsebreaker]

I had both already filled in unfortunately.

Situation remains the same, if I refresh the token manually on the Netatmo website and copy the new Refresh Token in the credentials file it will work but just 1 time.

Op ma 3 jun 2024 om 23:08 schreef rvk01 @.***>:

Just tried again, no changes. It did work, it doesn't work now.

Did you add the data protection officer name & email in the app registration page?

I just tried it with freshly created token via the web interface and it worked here. (used user root instead of normal pi to test)

— Reply to this email directly, view it on GitHub https://github.com/philippelt/netatmo-api-python/issues/82#issuecomment-2146126849, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF5ASQAYFN3IUQ4JZJQBTW3ZFTLORAVCNFSM6AAAAABIWZ2Y4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGEZDMOBUHE . You are receiving this because you authored the thread.Message ID: @.***>

[rvk01]

I had both already filled in unfortunately. Situation remains the same, if I refresh the token manually on the Netatmo website and copy the new Refresh Token in the credentials file it will work but just 1 time.

Did you check if the "CLIENT_ID" and "CLIENT_SECRET" are also exactly as stated on the App-page? For me, with the OAuth2 method, I used the wrong CLIENT_ID. In the past a different CLIENT_ID worked fine but it crapped out at 12:00 this afternoon.

[pulsebreaker]

Yes I did to be sure.

I copied the lnetatmo.py file from the github and pasted it directly in the site-packages directory on my pc. This seemed to change something (I'm not sure) because it looks like it works again. But when I update the credentials file on my NAS, which is where I want it to run, it works just 1 time. After that the script also doesn't work on my pc anymore.

This tells me that the credentials file on the NAS isn't updated with the new refresh token since this afternoon. It did before, strange.

Also funny, some dashboard data sometimes doesn't show up, but that's for later.

It's late, I'm going to bed and continue tomorrow.

Thanks for all the help.

Arno

Op ma 3 jun 2024 om 23:41 schreef rvk01 @.***>:

I had both already filled in unfortunately. Situation remains the same, if I refresh the token manually on the Netatmo website and copy the new Refresh Token in the credentials file it will work but just 1 time.

Did you check if the "CLIENT_ID" and "CLIENT_SECRET" are also exactly as stated on the App-page? For me, with the OAuth2 method, I used the wrong CLIENT_ID. In the past a different CLIENT_ID worked fine but it crapped out at 12:00 this afternoon.

— Reply to this email directly, view it on GitHub https://github.com/philippelt/netatmo-api-python/issues/82#issuecomment-2146173701, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF5ASQAANBFS67Z7Z76XUDDZFTPIJAVCNFSM6AAAAABIWZ2Y4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGE3TGNZQGE . You are receiving this because you authored the thread.Message ID: @.***>

[rvk01]

Yes I did to be sure. I copied the lnetatmo.py file from the github and pasted it directly in the site-packages directory on my pc. This seemed to change something (I'm not sure) because it looks like it works again. But when I update the credentials file on my NAS, which is where I want it to run, it works just 1 time.

That seems to indicate it is a local persmission problem. I also copied the lnetatmo.py because it wasn't installed under my root user (just to test) and it worked. But I just copied it next to my .py file. So if it works there on the pc, then it should also work in the NAS.

Are you sure the correct library for lnetatmo.py is loaded on the NAS? What are the permissions of the credential file? Are they correct for the user which python script is running under?

I noticed the library only stores the REFRESH_TOKEN, not the ACCESS_TOKEN. The ACCESS_TOKEN is valid for about 3 hours. So the library always retrieves a new ACCESS_TOKEN when started. If the corresponding REFRESH_TOKEN (which is also renewed) can't be stored in the credential file, the next run will fail with an old REFRESH_TOKEN.

[pulsebreaker]

Problem solved!

You were right, it was an issue with not writing the new Refresh Token to the credentials file on the Synology NAS. The file permissions were still good (unchanged) so that was strange.

I managed to get the newer lnetatmo.py file from here into the site-packages directory on the NAS (took some fiddling), checked all file permissions again and bingo, it started to work again.

I guess the version of lnetatmo I used was too old, something changed at the Netatmo site and it stopped working.

Thank you again for all the help.

[philippelt]

Good news @pulsebreaker , I am glad that authentication did not change again :wink:

[philippelt]

BREAKING NEWS ,

one user sent me today a response from Netatmo support about recent authentication issues. Here is a copy of this response :

Hello,

We just did a modification on the token retrieval process :

When you refreshed an access_token using the associated endpoint https://api.netatmo.com/oauth2/token, Netatmo servers responded with a couple of tokens : an access_token and a refresh_token. If the previous access_token was still valid, the refresh_token value was never renewed

Now, this behavior changed to be compliant with the recommendations of the RFC of the OAuth2 Authorization Framework (section 10.4) and improving the security of the data of our users When refreshing tokens, access_token and refresh_token values will be automatically renewed and former tokens invalidated

So, if you do not update and use the new refresh_token value when refreshing your access_token, your users will be disconnected after 3 hours and you will retrieve an “invalid_grant” error To fix it, you need to update the tokens value as soon as you get the newly generated ones

Si finally there was an effective change in the Netatmo authentication service. I don't know if this was announced/documented anywhere but Netatmo has a long tradition of breaking things that works :smile:

This can explain why we did have different behaviors exposed here. If you were using new token at each launch (like I do with cron starting my small monitor agent), you should not encounter problems. If you have monitor programs running permanently, that means calling Authentication only on startup, it will rely on management of access token expiration to get renewed.

Thus I have to check that, in that case, that the refresh token renewal is correctly handled.

[devMaFi]

On Monday at 12am my implementation stopped working. So I downloaded the latest version of the lnetamo.py file and give it a try...some issue as reported, one called worked, the following failed with the invalid grant error.

After changing to a credentials file with client id, client secret and refresh token I found at, that the refresh token is updated after each request.

Since I adapted this, the service is running continuously without issues.

May this helps the one or other

[LArtisanDuDev]

BREAKING NEWS ,

one user sent me today a response from Netatmo support about recent authentication issues. Here is a copy of this response :

Hello, We just did a modification on the token retrieval process : When you refreshed an access_token using the associated endpoint https://api.netatmo.com/oauth2/token, Netatmo servers responded with a couple of tokens : an access_token and a refresh_token. If the previous access_token was still valid, the refresh_token value was never renewed Now, this behavior changed to be compliant with the recommendations of the RFC of the OAuth2 Authorization Framework (section 10.4) and improving the security of the data of our users When refreshing tokens, access_token and refresh_token values will be automatically renewed and former tokens invalidated So, if you do not update and use the new refresh_token value when refreshing your access_token, your users will be disconnected after 3 hours and you will retrieve an “invalid_grant” error To fix it, you need to update the tokens value as soon as you get the newly generated ones

Si finally there was an effective change in the Netatmo authentication service. I don't know if this was announced/documented anywhere but Netatmo has a long tradition of breaking things that works 😄

This can explain why we did have different behaviors exposed here. If you were using new token at each launch (like I do with cron starting my small monitor agent), you should not encounter problems. If you have monitor programs running permanently, that means calling Authentication only on startup, it will rely on management of access token expiration to get renewed.

Thus I have to check that, in that case, that the refresh token renewal is correctly handled.

Thanks for these news. I've done api call in C++ on ESP32 and my authent is broken since couple of days. Netatmo did it again !!

[rvk01]

The break from monday also had another factor. I wrongly used a different client_id with my access and refresh tokens. It was generated with the correct one but after that, the client_id wasn't checked any more (and I used another one).

As of monday 12:00 CEST netatmo implemented a change that client_id is also checked (and needs to be identical to the one with which the refresh_token was created). I found this out with my own PHP implementation.

@philippelt I still think the access_token and maybe time valid should be stored in de credentials file (besides the refres_token). access_token is normally 3 hours valid and you don't need to throw it away each time you end the script. If you keep the script running for a long time, this doesn't matter but if you run the script every 10 minutes, there would be no need to get a new access_token. Netatmo doesn't penalize this yet but that could change in the future (getting new access_tokens while the previous one is still valid).

Only if using the access_token and getting an invalid error due to timeout of 3 hours, you can retrieve a new access_token and refresh_token. That's how I do it in PHP. I save the complete response from the retrieval of the access_token and reuse it until it errors out (after which I get a new one).

Maybe something for your todo-list 😁

[snikula]

Hi, im running simple test using cron, because my python-scrip which is used to retrieve my house temperature does not work after Netatmo change something? Test-script:

import lnetatmo from datetime import datetime print(datetime.now()) f = open("/home/sami/test.txt", "w") f.write(str(datetime.now())) f.close() authorization = lnetatmo.ClientAuth() devList = lnetatmo.WeatherStationData(authorization) print (devList.lastData()['Ulkolämpötila']['Temperature'])

Done yesterday:

1. 07:27 authorize app after generating new Refresh Token (https://dev.netatmo.com/)
2. Writing new Refresh token in .netatmo.credentials (its writable to test.py:  -rwxrwxrwx 1 root root 189  6. 6. 07:28 .netatmo.credentials)
3. First run: 2024-06-06 07:29:38.117679
4. New Refresh token is writen to .netatmo.credentials in every test.py run
5. After ~3h it stops working
6. Trying to run manually to get error code, result: code=400, reason=, body=b'{"error":"invalid_grant"}' Traceback (most recent call last):   File "/home/sami/test.py", line 15, in     devList = lnetatmo.WeatherStationData(authorization)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/home/sami/.local/lib/python3.11/site-packages/lnetatmo.py", line 413, in init     self.getAuthToken = authData.accessToken                         ^^^^^^^^^^^^^^^^^^^^   File "/home/sami/.local/lib/python3.11/site-packages/lnetatmo.py", line 240, in accessToken     if self.expiration < time.time() : self.renew_token()                                        ^^^^^^^^^^^^^^^^^^   File "/home/sami/.local/lib/python3.11/site-packages/lnetatmo.py", line 251, in renew_token     if self.refreshToken != resp['refresh_token']:                             ~~~~^^^^^^^^^^^^^^^^^ If I generate new Refresh token in dev.netatmo and writing it to .netatmo.credentials script will work again ~3h.

Any ideas?

[mika1337]

Hi all.

I have done some test in a fork of this library and a solution that seems to work is to:

• store the access_token in credentials file
• if the token hasn't expire, use directly the access_token
• upon token expiration, update access_token and refresh_token in the credentials file

[rvk01]

Yes, that's how I do it.

• if the token hasn't expire, use directly the access_token

Only thing is that I don't store the expiration date. I just always try the access_token and if I get a error.code == 3 then the access_token is expired and I need to get a new one.

But I also have script 4.0.0 running here, and that one still works.

[rvk01]

1. 07:27 authorize app after generating new Refresh Token (https://dev.netatmo.com/)
2. Writing new Refresh token in .netatmo.credentials (its writable to test.py:  -rwxrwxrwx 1 root root 189  6. 6. 07:28 .netatmo.credentials)
3. First run: 2024-06-06 07:29:38.117679
4. New Refresh token is writen to .netatmo.credentials in every test.py run
5. After ~3h it stops working

I first thought it could be something with self.expiration (because there is potentially a problem on exact second of expiration). But that only happens if you keep your script running.

You start the script each time again and again. Then the access- and refresh-tokens should be refreshed each time.

• Please check if you are using the latest lnetatmo.py (the linenumbers in your error seem to be different from latest source).
• And also please check if you have the same CLIENT_ID in your credential file as in the web-site for the app at dev.netatmo.com.

[snikula]

Thanks for help! CLIENT_ID is same Where to find latest release? Im trying wget https://github.com/philippelt/netatmo-api-python/archive/v4.1.0.tar.gz, (from setup.py) ERROR 404: Not Found.

• Please check if you are using the latest lnetatmo.py (the linenumbers in your error seem to be different from latest source).
• And also please check if you have the same CLIENT_ID in your credential file as in the web-site for the app at dev.netatmo.com.

[rvk01]

Thanks for help! CLIENT_ID is same Where to find latest release? Im trying wget https://github.com/philippelt/netatmo-api-python/archive/v4.1.0.tar.gz, (from setup.py) ERROR 404: Not Found.

You can click code above at the left and click code again at the right (green button) then download as .zip. It's this link https://github.com/philippelt/netatmo-api-python/archive/refs/heads/master.zip

It will give you the latest from this repository. (Not sure what the other link points at)

I'm not sure if pip also has the the latest.

[philippelt]

Hello all,

In fact, the netatmo change should not impact existing code. It's right that renewing access token at each run is more than strictly required but it should not create any problem.

I wonder if there can be an issue with the home relative path of the credential file.

for people who still have problems, would you please pick directly from github the last release 4.1.0 and add to your ClientAuth call the credentialFile parameter like the following example :

authorization = lnetatmo.ClientAuth(credentialFile="/my/absolute/path/to/.netatmo.credentials")

[snikula]

https://github.com/philippelt/netatmo-api-python/archive/refs/heads/master.zip Thanks! Ill try this version.

[rvk01]

In fact, the netatmo change should not impact existing code. It's right that renewing access token at each run is more than strictly required but it should not create any problem.

No, it's not at the moment. Until netatmo decides to give an error if you use a refresh_token under which there is still a valid access_token. If I would implemented this and don't want to burden the system too much, I would have done that.

But ok, netatmo still allows using a refresh_token even when the access_token is still valid (and they choose to invalidate the old refresh_token instead) so there is no problem (yet) 😉

[philippelt]

I understand your point @rvk01 but as Netatmo declared in their response, they want to comply with the RFC of the OAuth2 Authorization Framework and there is no such behavior described in the specs.

You can always loose an access token, if your program encounter a bug and fail for example. If you can't obtain a new token from the refresh token, you would be obliged to wait for 3 hours before being able to recover or restart from scratch by creating a new refresh token.

By the way, your suggestion to save the access token to be able to reuse it while it is still valid can be a good idea. I can't remember for the moment why I decided not to save it. I Think it was because I had to different servers asking for data with the same refresh token in the past (when refresh token did last forever) and I was afraid of side effects of access token invalidation by one server where the other was trying to use it.

I am not yet sure that our current problems are related to the access token. Strangely, I have been able to reproduce the problem using cron tasks and the solution was to explicitly mention the credential file path. It is like if expanduser("~/.netatmo.credentials") was no longer working for cron jobs. The current issue may be the consequence of python/OS upgrades (expanduser implementation changed in python 3.10.2) from our servers rather than Netatmo servers for which, as I said, the change should not break existing code.

In this case, I should try to use Path.home() and Path module methods instead of simply os.path.expanduser() the relative file path.

[rvk01]

By the way, your suggestion to save the access token to be able to reuse it while it is still valid can be a good idea. I can't remember for the moment why I decided not to save it. I Think it was because I had to different servers asking for data with the same refresh token in the past (when refresh token did last forever) and I was afraid of side effects of access token invalidation by one server where the other was trying to use it.

No, server etc. were always the same. As I can remember you didn't implement this because you always had use-cases where the script was long-running. In which case the access_token was reused because it was in memory. You never had need for starting the script via cron every 10 minutes or so. That's why, I think, you didn't implement the saving of the access_token (we had a discussion about this in another issue in the past).

I am not yet sure that our current problems are related to the access token. Strangely, I have been able to reproduce the problem using cron tasks and the solution was to explicitly mention the credential file path. It is like if expanduser("~/.netatmo.credentials") was no longer working for cron jobs.

For me it still works fine (python 3.11.2) as user pi.

BTW. I already mentioned, but there might be a small time critical issue when you have a long running script. You save self.expiration in the script as time+expires_in. But this could be a second or so off in which case the access_token was expired. For example if the accesstoken() is called one second before the time expires (but the server at netatmo sees it as such).

So you might want to build in a few seconds of margin in this line (substract 10 seconds from self.expiration for example): if self.expiration < time.time() : self.renew_token()

Because if the access_token is expired... I also don't see any fallback after each communication to getting a new one in the code when the access_token is invalid... or am I missing something? Not the issue above... but I thought I'd mention it anyway.

[pulsebreaker]

Following this discussion closely :-)

In my case the script on the NAS is started and ended every 15 minutes (because you get new data from Netatmo only every 10 minutes or so?), it updates the Refresh Token in the credentials file each run and is still working perfectly since 4 days ago.

[trygveasp]

I saw this issue first now when I wanted to report my findings. I was able to fix my code, but I have to make sure I read the REFRESH_TOKEN from the credential file and construct the ClientAuth object with refreshToken as an argument.

        if os.path.exists(credentials):
            with open(credentials, mode="r") as fh:
                refreshToken = json.load(fh)["REFRESH_TOKEN"]
            authorization = lnetatmo.ClientAuth(refreshToken=refreshToken)
        else:
            authorization = lnetatmo.ClientAuth()

The problem is that unless this is specified it will always use the one from when lnetatmo was imported and that is normally not what you want if you like me have a real-time updating weather station display. So the lnetatmo code is not really broken, but I am not so sure the default refreshToken int ClientAuth should be the one from when the library was imported (_REFRESH_TOKEN)

[rvk01]

I saw this issue first now when I wanted to report my findings. I was able to fix my code, but I have to make sure I read the REFRESH_TOKEN from the credential file and construct the ClientAuth object with refreshToken as an argument.

That's strange because that's exactly what the current code in the library does. Are you sure you have the latest version?

[trygveasp]

I saw this issue first now when I wanted to report my findings. I was able to fix my code, but I have to make sure I read the REFRESH_TOKEN from the credential file and construct the ClientAuth object with refreshToken as an argument.

That's strange because that's exactly what the current code in the library does. Are you sure you have the latest version?

I use 4.0.0 installed from pypi with pip. The problem with that version is that although the refresh token is renewed and the data is updated the first time, the default constructor always uses the original token from the import and thus throwing invalid grant.

[rvk01]

I use 4.0.0 installed from pypi with pip. The problem with that version is that although the refresh token is renewed and the data is updated the first time, the default constructor always uses the original token from the import and thus throwing invalid grant.

That's strange. I also use 4.0.0 here (which is not the latest) and for me it always picks the correct one from the credential file.

Are you sure there are no environment settings set for refresh_token. When they are, they are preferred over the credential file, I think.

You say 'from the import'. What do you mean by that. The refresh_token should (and is) always taken from the credential file (unless specifically passes in the function call, which shouldn't be necessary if the correct one is in the credential file.

So just set it manually the first time in the credential file (make sure the permissions are still correct) and call the function without parameters.

[trygveasp]

I use no enviroment variables any longer. And you are right, the first time the refresh token is read from the credential file in a variable which is set when the lnetatmo module is imported in my script. This variable is used as the default argument in the constructor and is never updated (at least in my case) even though the refresh token is updated in the credential file.

[rvk01]

This variable is used as the default argument in the constructor and is never updated (at least in my case) even though the refresh token is updated in the credential file.

It (self.refreshtoken) is updated in def renew_token(self) (when retrieving the access_tokeb). At least in the latest code. Also there on line 252 of version 4.0.0 too.

[trygveasp]

An example to demonstrate how it fails:

import lnetatmo
import time

def get_measurement():
    try:
        authorization = lnetatmo.ClientAuth()
        weather_data = lnetatmo.WeatherStationData(authorization)
    except:
        raise Exception

while True:
   print("get_measurement")
   get_measurement()
   time.sleep(1)

The first construction of ClientAuth is fine. The first call to WeatherStationData is fine but triggers an update of REFRESH_TOKEN and the ClientAuth object is updated accordingly and also the ~/.netatmo.credentials file. So far all good. However, in next iteration the ClientAuth still gets the REFRESH_TOKEN from the initial call to ClientAuth and the raise an invalid_grant exception. I guess many use this kind of workflow.

[rvk01]

However, in next iteration the ClientAuth still gets the REFRESH_TOKEN from the initial call to ClientAuth and the raise an invalid_grant exception. I guess many use this kind of workflow.

How are you determining this? With a long running script (like you are showing) the internal self.refrestoken should be used (at least in latest 4.1.0). It shouldn't take any old REFRESH_TOKEN. Where in the code do you see that it takes the old one?

Anyway, for long running script you definitely should take the last version 4.1.0. Not 4.0.0. That version might not work for long running scripts. Only for short running cron scripts.

[trygveasp]

However, in next iteration the ClientAuth still gets the REFRESH_TOKEN from the initial call to ClientAuth and the raise an invalid_grant exception. I guess many use this kind of workflow.

How are you determining this? With a long running script (like you are showing) the internal self.refrestoken should be used (at least in latest 4.1.0). It shouldn't take any old REFRESH_TOKEN. Where in the code do you see that it takes the old one?

Anyway, for long running script you definitely should take the last version 4.1.0. Not 4.0.0. That version might not work for long running scripts. Only for short running cron scripts.

I know it because I print the params to postRequest and I see that refreshToken never changes even though it has been renewed, updated and the crediential file is updated:

def postRequest(topic, url, params=None, timeout=10):
    if PYTHON3:
        req = urllib.request.Request(url)
        if params:
            req.add_header("Content-Type","application/x-www-form-urlencoded;charset=utf-8")
            if "access_token" in params:
                req.add_header("Authorization","Bearer %s" % params.pop("access_token"))
            params = urllib.parse.urlencode(params).encode('utf-8')
        try:
            print("params", params)
            resp = urllib.request.urlopen(req, params, timeout=timeout) if params else urllib.request.urlopen(req, timeout=timeout)

My code was running fine with 4.0.0 in my long running script until last week when netatmo changed their API. I can test 4.1.0, but there is no package on pypi and no tag or release in this repo.... So where do I find 4.1.0?

[rvk01]

My code was running fine with 4.0.0 in my long running script until last week when netatmo changed their API. I can test 4.1.0, but there is no package on pypi and no tag or release in this repo.... So where do I find 4.1.0?

You can click code above at the left and click code again at the right (green button) then download as .zip. It's this link https://github.com/philippelt/netatmo-api-python/archive/refs/heads/master.zip

It will give you the latest from this repository. I'm not sure why pip doesn't have the latest.

[snikula]

Thanks for help! CLIENT_ID is same Where to find latest release? Im trying wget https://github.com/philippelt/netatmo-api-python/archive/v4.1.0.tar.gz, (from setup.py) ERROR 404: Not Found.

I'm not sure if pip also has the the latest.

Thanks! My script (and test-script) is now running with 4.1.0 with no errors.

[philippelt]

OK, there are troubles on Netatmo servers. To clarify the situation, I created a small script that just authent periodically (every 30 min) with always the same credential file.

Today at 5:02 UTC, I log this error :

Your current token scope do not allow access to Weather station

Yesterday at 7:02 UTC

code=400, reason=, body=b'{"error":"invalid_grant"}'

At jun 6, 5:02 UTC

Your current token scope do not allow access to Weather station

And the script is currently still running fine every 30 min without any change whether in the code or the .netatmo.credentials file.

Obviously there is an intermittent problem on Netatmo side. I think that the 4.1.0 version of the library should work without problem.

I will keep my script running to continue monitoring of Netatmo servers. I will publish the 4.1.0 to Pypi in a few days if I don't discover anything new. In the meantime, juste download the repo copy of lnetatmo.py to test with you application.

[trygveasp]

You can click code above at the left and click code again at the right (green button) then download as .zip. It's this link https://github.com/philippelt/netatmo-api-python/archive/refs/heads/master.zip

It will give you the latest from this repository. I'm not sure why pip doesn't have the latest.

I can confirm that both my original code and the test I provided works fine with master! Would be nice to have a release of 4.1.0 on github/pip instead of downloading master;-)

[buldre]

I got the same issues as described above yesterday and today with 4.0.0 on a Raspberry Pi 4 running Python 3.11. I refreshed the Refresh Token and changed the permission on the credentials file. The file was then updated on each code start with a new Refresh Token. Then it worked once with a new Refresh Token, and now it always fails, ant the credentials file does not update.

I then tried 4.1.0 which also fails and does not update the file.

Here are two runs with 4.0.0 and 4.1.0: Owner : user Group : user View : Anybody Change : Only owner Execute: Nobody Note = New Refresh Token. File NOT overwritten.

%Run fnct_verify_Netatmo_connected.py Printing Summary of Netatmo connection verification - Ver. 0 2024-06-21 15:12:19

code=400, reason=, body=b'{"error":"invalid_grant"}' ERROR:root:Exception occurred Traceback (most recent call last): File "/home/user/PythonScripts/ECfan/fnct_verify_Netatmo_connected.py", line 29, in verifyConnected weatherData = lnetatmo.WeatherStationData(authorization) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/PythonScripts/.venv/lib/python3.11/site-packages/lnetatmo.py", line 413, in init self.getAuthToken = authData.accessToken ^^^^^^^^^^^^^^^^^^^^ File "/home/user/PythonScripts/.venv/lib/python3.11/site-packages/lnetatmo.py", line 240, in accessToken if self.expiration < time.time() : self.renew_token() ^^^^^^^^^^^^^^^^^^ File "/home/user/PythonScripts/.venv/lib/python3.11/site-packages/lnetatmo.py", line 251, in renew_token if self.refreshToken != resp['refresh_token']:

TypeError: 'NoneType' object is not subscriptable
e: 'NoneType' object is not subscriptable
>>> 

Permissions = Same as above
Note = Same Refresh Token as earlier today which has not been overwritten. File NOT overwritten in this test either
>>> %Run fnct_verify_Netatmo_connected.py
Printing Summary of Netatmo connection verification - Ver. 0
2024-06-21 20:01:43

auth...= <lnetatmo.ClientAuth object at 0x7fb0638c10>
code=400, reason=, body=b'{"error":"invalid_grant"}'
ERROR:root:Exception occurred
Traceback (most recent call last):
  File "/home/user/PythonScripts/ECfan/fnct_verify_Netatmo_connected.py", line 29, in verifyConnected
    weatherData = lnetatmo.WeatherStationData(authorization)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PythonScripts/.venv/lib/python3.11/site-packages/lnetatmo.py", line 416, in __init__
    self.getAuthToken = authData.accessToken
                        ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/PythonScripts/.venv/lib/python3.11/site-packages/lnetatmo.py", line 241, in accessToken
    if self.expiration < time.time() : self.renew_token()
                                       ^^^^^^^^^^^^^^^^^^
  File "/home/user/PythonScripts/.venv/lib/python3.11/site-packages/lnetatmo.py", line 252, in renew_token
    if self.refreshToken != resp['refresh_token']:
                            ~~~~^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
e: 'NoneType' object is not subscriptable
>>> 

I tried 4.1.0 another time with a new Refresh Token and permissions 
View   : Anybody
Change : Anybody
Execute: Nobody
It worked once and the file was overwritten, then it failed again.

I noted however, when I clicked to copy the tokens on my Netatmo page in Safari and pasted them into a Notes document on my Mac that some capital letters pasted as small caps, so beware! Better to read the Tokens off the Netatmo page and take care to check capital O versus zeros and small caps L versus capital I.