Netatmo API authentication update

[pzystorm]

Some days ago I have received a mail from Netatmo that they will remove the authentication via username and password from the API. From Oktober 2022 there will be only the oauth2 process.

Question to @philippelt and all users of the netatmo-api-python: What do we need to do? Is it possible to patch the code to support oauth2?

[rnsc]

Hey @pzystorm , If you read the code you'll see that the library creates an oAuth2 token on the fly with the required permissions, so this shouldn't be an issue.

EDIT: I'm not the maintainer, but I use this library and just re-read the code to confirm this after receiving the email

[pzystorm]

Hi @rnsc thanks for your reply. Together we hopefully will figure out. I am not so familar with the oauth2 process, but if I look into here: https://github.com/philippelt/netatmo-api-python/blob/master/lnetatmo.py#L94 Yes, you are true, the oauth API endpoint is used. But if I then look at https://github.com/philippelt/netatmo-api-python/blob/master/lnetatmo.py#L194 the grant_type is password, which means, that the code itself knows username and password and sends them directly to the netatmo API. If I read Netatmo's mail correctly this is exact the procedure which will be revoked. As far as I know oauth2 should be something like: the code opens the browser (or on a remote server: it gives you an url which you need to open locally) then the browser displays some netatmo login page, you enter your username/password there and then the browser closes and some kind of token is handed back to the code. I am missing this kind of flow in the code. Do you have any clue on this?

[rnsc]

@pzystorm Ah! I see, indeed, I skipped over the password part. I'll investigate this weekend if I can update the code in a fork as I'll need to have this working for my own needs anyway 😅

[rnsc]

Ok, so this makes it way less convenient, but we can receive a refresh token with the authorisation, meaning that the application needs to save both locally and be executed frequently enough to renew the token before it expires. Given the value specified in the documentation, it looks like it would need to be refreshed every 3 hours minimum.

[philippelt]

Hello,

you are right, this may become an issue for some. The initial intent of the application was to allow someone to monitor (using a custom software or other open source app) his own device. This is generally requiring automated access.

The idea is that anyone with such need (which is in fact an "unattended access") can not interact with the "user browser" to request authorization at access time.

There are two solutions here :

• The existing auth mechanism, used by the lnetatmo module, with the grant_type=password will continue to work without changes if the application credentials that you are using have been created in your account. This should be the case of most peoples and it's easy to create such credentials (check at https://dev.netatmo.com/apidocumentation/oauth).

• For unattended access, almost all online services using oauth2 mechanism allows a user, from the account security web page, to create access token by selecting the roles that you want to allow an app and the time you allow for the duration of the token (it frequently can be unlimited).

I understand the need for Netatmo to improve account security, which was not a strong requirement when they were just selling high-tech toys (weather station for examples) but become critical when you plan to control a lot of your home automation.

This move is a logical move. However, to avoid restricting usages in the futur, their would be an urgent need for Netatmo to provide the ability to create and manage long-lived application tokens in user accounts.

In the meantime, using app credentials created using the user account will be the uniq way to continue unattended uses.

At least, as long as Netatmo do not change this rule... 😇

Phil

[rnsc]

Ok, so I created an application in my account and got a ClientID and ClientSecret, that should be sufficient and be supported by the API for now then.

Thank you @philippelt for taking the time to check this one out!

[macsakow]

we can receive a refresh token with the authorisation, meaning that the application needs to save both locally and be executed frequently enough to renew the token before it expires. Given the value specified in the documentation, it looks like it would need to be refreshed every 3 hours minimum.

Actually, the refresh token remains valid for way longer than 3 hours. I don't know how long exactly, but I've managed to refresh access tokens which had been expired for more than 20 hours.

[trho]

we can receive a refresh token with the authorisation, meaning that the application needs to save both locally and be executed frequently enough to renew the token before it expires. Given the value specified in the documentation, it looks like it would need to be refreshed every 3 hours minimum.

Actually, the refresh token remains valid for way longer than 3 hours. I don't know how long exactly, but I've managed to refresh access tokens which had been expired for more than 20 hours.

I am also working on the issue, because my home automation is based on this library and I got alarmed by the mail from netatmo. AFAIK, the refresh token does not expire, but netatmo might change this in future, see also answers from netatmo support https://github.com/ssadams11/node-red-contrib-netatmo/issues/27#issuecomment-1214161943.

[trho]

I contacted the netatmo support concerning the exception you mentioned in https://github.com/philippelt/netatmo-api-python/issues/46#issuecomment-1207245040.

Q: In your Netatmo API Security Update Mail you mention that the Client Credentials grant type method will be completely removed in October. But in the documentation (https://dev.netatmo.com/apidocumentation/oauth#client-credential) an exception is mentioned: "This method can only be used with the same account that the one who owns the API application. If you want to access data from another user's account, you MUST use the Authorization code grant type." Will this exception be removed in October, too?

A: Indeed, the goal is to totally remove this authentication method end of October. So, it will not possible at all anymore to use username/password for authorize requests (client credentials grant type) and it will be mandatory to use Authorization code grant type

I made a pull request (https://github.com/philippelt/netatmo-api-python/pull/48) that handles the creation of the refresh_token as part of the main function if the refresh token has not been created and store it in the .netatmo.credentials file. I also added a property "SCOPE" to configure the needed scopes in the credentials file. For more details see the PR description.

Only restriction right now - the http server is Python3 only. I could add that if you are interested in the PR.

[philippelt]

Hello trho,

many thanks to investigate this problem. You mentionned that Netatmo said that the password authentication method will also be removed end of october, there is no mention of this in the documentation.

I have seen many breaking changes and sudden U-turns from Netatmo dev team but I hope they will not be stupid enough to take such decision without previous announcement and providing an application token solution for unattended access.

Regarding your proposal, I see several issues :

• I hate breaking changes, I like things that continue to work even when the world change. As this library may have been embedded in small devices that may not be up to date, I would prefer to continue to support python 2. We could start to mention deprecation of python 2 but this could not be for end of october, we need a longer delay.

• It is not possible to start a http listener for such use. Most users are on home lan and have NATed addresses and a gateway that must explicitly allow port forwarding to listen for a new service. For security reason, I do not want to use UPNP or ask user to open new port for this use. With the progressive adoption of IPv6 where NAT protection disappear, some users may forget to enable a firewall to protect their devices and we should not listen to any port without users been aware of.

• I would like the library to continue to work the same way (did I said that I hate breaking changes ?). I agree with you that we could add a refresh token to the credentials, with respect to the existing policy (REFRESH_TOKEN in the credential file, in an env var or as an explicit parameter at call). The rule would become to use first refresh token if available and if absent or failure to fallback to user/password. In this way, as long as Netatmo respect their announcement, the library will continue to work without changes.

• It would be necessary to provide a companion utility to get the refresh token. Printing a URL that the user can copy to a browser to grant the access and then using the API to retrieve the refresh token and add it in the .netatmo_credentials file would be nice (no need to listen http): this is the method used by most services I use, some like google or github allow even simpler method with application password. May be Netatmo consider that their needs is above such levels.

I think it's better to see what Netatmo will actually do, as it may not be what they announced, before deciding a move. If they really totally disable non interactive authentication which is not what they currently announce, this may be a try to discourage to use unattended access, may be to reduce the load on their servers. I think that many users would have serious problems.

[trho]

I will reference this thread in the communication with Netatmo and ask if they can add the removal notice also in the documentation and come back to you.

About Python 2 support: I could easily add this, it is only about providing a local Python 2 http server. I get you concerns about opening a port on embedded devices, a separate utility would not be much effort, too. Basically this is already implemented this in the main module, including writing the token to the config file. You can start the module e.g. on your laptop with "python -m lnetatmo" and copy the local config file to the target device. This is what I have done in my local setup, too. I am wondering if this isn't already sufficient.

[trho]

Answer from Netatmo Support: "I asked the developers to add a line in the documentation to talk about the deprecation of the method. But I don't know when it will be done. For information, this change is done to enhance the security of third-party access Also, we don't want to provide tokens with an increased lifespan for the same security reasons"

[pzystorm]

Big thanks for the change regarding oauth2. It works pretty good. I have re-added the python2 compability and added some more lines regarding pretty print of the ~/.netatmo.credentials file and some more lines when running "python -m lnetatmo" Commit: see above

[Boulder08]

There was another message from Legrand notifying changes in the authentication on April 17th. Does this mean that the username-password -combination used by this module will stop working? I thought that was supposed to happen already earlier but everything seems to work fine for now.

[philippelt]

Hello, the coming update will only imply that the existing access tokens of an application will systematically expire after 3 hours whatever happen.

This would be a problem if you are using very long lived sessions, but as far as I understand this change, it will have no impact on existing usage if your session duration (the time during which you will use tokens obtains using ClientAuth) is less than 3 hours.

In other cases, you should ensure to refresh your ClientAuth data periodically.

[philippelt]

Finally, it seems that today Netatmo decided to stop entire support of grant_type "password".

Fortunately, it is possible to create an access_token/refresh_token in the App management interface (Token generator in https://dev.netatmo.com -> MyApps -> Token generator). Select required scope, generate token and replace USERNAME and PASSWORD fields with REFRESH_TOKEN field.

The lnetatmo.py library has to be updated to v3.0.0 (available in pypi). Let me know if you encounter problems

[Boulder08]

Thank you, seems to work well. I had some really old code in my home automation script but managed to switch to the new system (finally).

Does this now mean that in case the script doesn't run often enough, the token will expire and I have to renew it and adapt the script accordingly?

[rvk01]

Does this now mean that in case the script doesn't run often enough, the token will expire and I have to renew it and adapt the script accordingly?

No, the refresh_token will not expire. It is used to get an access_token. The access_token will expire in which case the refresh_token is used again to get a new access_token. I actually don't now if this code will get an new access_token EACH time (which shouldn't be happening because it should reuse the old access_token until it is really expired in about 3 hours).

(Edit: The reusing of the access_token probably only works as long as you have your script running. Exiting your script and starting it again will trigger a retrieval of a new access_token. It would be better if the access_token was also saved to a file but maybe you could do in your own code. But I don't know if there is an option to provide a saved access_token, I only see the refresh_token).

I also would have preferred code which retrieves a refresh_token according a authentication screen, which would allow OTHER users with other devices to use this code (so they don't need to set their own app-code etc.). I use a similar technique with a PHP script. But this works OK too, if the user/programmer owns the device (s)he wants to read.

[philippelt]

Obviously, it must work or else I would have already be notified of troubles. Closing until next Netatmo breaking change :slightly_smiling_face: