No support for username/password from October ‘22

[SnorreSelmer]

I got this email yesterday:

To improve the security of our products, we inform you that the Client Credentials grant type method will be completely removed. It will no longer be possible to authenticate with the username and password of the user. 

The effective date of this update is October 2022.

How can you authenticate with Netatmo API ?
From this date, the OAuth2 authorization code flow must be followed for authentication.

Is this exporter ready for this change? What needs to be altered in the compose-file?

[xperimental]

I got this email today as well, as I have not read through the documentation yet, I can't say what the changes will be.

I would assume that this means changing how the exporter does authentication and will result in code and configuration changes. There's still some time, though :slightly_smiling_face:

[SnorreSelmer]

Excellent! Good to see you’re aware! I love this exporter.

[mpitt]

Any news? October is getting closer… 😅

[theonlydoo]

Netatmo seems to have given up on updates, they don't support their own 2FA

[xperimental]

Any news? October is getting closer… :sweat_smile:

I did realize this last weekend as well and started looking into it. Need to update the netatmo-api-go library as well. I think I have an idea on how to approach this in a user-friendly way, but overall the authentication flow is not really suited for server-side software unless you have some central provisioning which does not really match the purpose of this project.

[mpitt]

overall the authentication flow is not really suited for server-side software

Yes that was my impression too, unfortunately. Thanks for the update and for your work!

[xperimental]

I plan to have two options:

• a "companion mode" where you do the login flow on another device and then copy the token into the exporter's configuration
• an "interactive mode" where the exporter provides a web-UI to do the normal interactive flow

There are "ideas" right now, nothing implemented yet though, so things might change. I also do not know how long the authentication will be valid and how long the exporter would be able to extend/refresh the token. The exporter might also be able to store the token persistently somewhere.

[xperimental]

I have just realized that I (for whatever reason :shrug: ) misinterpreted the date as "the 22nd of October" and not "start of October 2022". Sorry for that, let's see if I can make it work this week :slightly_smiling_face:

[xperimental]

I have a "first draft" ready. It's a bit cumbersome to use, mostly depending on "where" your exporter is running. I think I'll be adding the "companion mode" as well, but for now the exporter is "interactive".

The work-in-progress is in the new-auth branch (working together with a branch of the same name in my netatmo-api-go fork). I have built an image from the current state, which is available as ghcr.io/xperimental/netatmo-exporter:new-auth.

This removes the username and password from the configuration and starts up the exporter "unauthenticated". You need to visit the /authorize path on your exporter for the authentication process to start. It should redirect you to the NetAtmo page asking you to authenticate to the exporter application you registered.

There are two new configuration parameters:

• external-url, which controls the URL the NetAtmo Interface will redirect you back to after authenticating. This defaults to http://127.0.0.1:9210 which only works if the exporter is running "locally", so you probably need to change this.
• token-file, which, if not empty, makes the exporter save the current token on exit and load it on startup if the file exists.

Theoretically it should be possible to stop/start the exporter with the token saved to the token-file as long as the token is not expired. This is something I want to test myself during the next days.

Please try out the work-in-progress version. Thanks for your feedback.

[xperimental]

Today's update is the "authentication companion", a webserver displaying a very simple web frontend for authorizing against the NetAtmo API and getting a token.json file that can be loaded into netatmo-exporter. This is in the auth-tool branch based on yesterday's branch.

You have to compile this yourself (clone, checkout the branch, then do make auth-tool) and then run it using:

./auth-tool --client-id $YOUR_CLIENT_ID --client-secret $YOUR_CLIENT_SECRET

It will display a local URL to open in your browser which contains additional instructions.

Good luck! :four_leaf_clover:

[xperimental]

Apologies for the missing updates, but I am currently dealing with a non-digital emergency. The image mentioned in the earlier comment is running fine for me since posting the comment, so it seems to be fine, just needs a bit of cleanup and documentation.

It's possible that exporters that are running will keep running until the token they use finally expires.

[alvaroaleman]

Hey @xperimental thanks for your awesome work on this exporter!

I just wanted to mention that on the app page in the netamo developer portal, there is a "Create Token" button which can be used to generate a token. Using the new-auth tag and adding the --token-file flag pointing to a file whose content is {"access_token":"$TOKEN"} worked perfectly for me and might avoid the need of implementing the login flow in the exporter.

[xperimental]

@alvaroaleman I noticed that button as well, but it only produces an access_token with no refresh-token. That way, once the lifetime of the access-token has expired, the exporter has no way of renewing its token automatically.

The tokens produced by the auth-tool or the new version of the exporter also contain a refresh-token, so they can be renewed before their original lifetime expires (three hours, if I remember correctly).

Please test with the website-generated token on your end and report what lifetime the token has.

[alvaroaleman]

@xperimental oh interesting. Their website doesn't mention about an expiry nor does it show a refresh token. Also, the request to get the actual token that is done by their website gets redirected to another endpoint that returns a bunch of HTML with the token embedded.

I also manually queried the api with curl in the hope that they send any useful response headers back but no luck.

I'll keep it running with this token for now and will report back when/if I get authentication errors. Thanks!

[alvaroaleman]

So the website-generated token stopped working after about six hours :upside_down_face: I've generated one including refresh token from the auth-tool and that appears to be working fine so far :)

[j7lb5r]

Hello there. I found a tutorial how to include my netatmo in a docker container for unraid, but it based on version 1.3.0 :(. After two days of horror with " level=fatal msg="Error in configuration: need a NetAtmo client ID"" I found this thread. I am sorry I am not expert for this, is there any step by step manual how to set up it with new:auth? I generated the Access and refresh token -and now? THX

[xperimental]

@j7lb5r Unfortunately I have not yet written the instructions for any of the new login flows. How did you get the access-token/refresh-token? Using the auth-tool or using the authentication integrated into the exporter?

The content in this PR is also not yet merged into the master branch or released. I'll try to do this as soon as possible, maybe I have some free cycles during PromCon :wink:

[j7lb5r]

I created both tokens at Netatmo Connect Dev Area. I found it below the Client ID generator. There can you generate both tokens.

[xperimental]

@j7lb5r Just checked, NetAtmo has updated their web-UI to also provide a refresh-token not only an access-token. That does make things easier (not as good as before, but easier than it was last time I checked). I do not know though, how the OAuth implementation in the exporter handles the missing expiration time. Best case, it immediately uses the refresh-token to get a new token with a known expiry and save that one. Worst case, it just expires like before.

This might be interesting information for you as well, @alvaroaleman :slightly_smiling_face:

[sbrooke]

Failed for me just yesterday. I'm running the build on RPi, not the docker. I see the new release is probably coming any day now so I'll wait. Just letting you know.

Thanks!

[xperimental]

@sbrooke Again, apologies for the delay. The release will be coming soon. :crossed_fingers:

If you're not using Docker, you should be able to build it yourself from the new-auth or auth-tool branch.

[sbrooke]

Cool! Now I just need to learn how to do that! :)

[sbrooke]

You have this up above:

_There are two new configuration parameters:

external-url, which controls the URL the NetAtmo Interface will redirect you back to after authenticating. This defaults to http://127.0.0.1:9210 which only works if the exporter is running "locally", so you probably need to change this. token-file, which, if not empty, makes the exporter save the current token on exit and load it on startup if the file exists._

How do I change the url? I've tried a couple of things and they aren't working.

[sbrooke]

Curl from the same system works, just can't access it from somewhere else.

[xperimental]

Just released version 1.5.0 to push out all the non-authentication-related changes. Interestingly the old authentication still seems to work fine for me, even though it should have stopped working more than a month ago. Can anyone confirm?

For testing with the new authentication, I have updated the "new-auth" branch with the released changes. The updated Docker image is this one:

ghcr.io/xperimental/netatmo-exporter:d59fbff-new-auth

[xperimental]

@sbrooke You should be able to set the "external-url" using either the --external-url command-line flag or the NETATMO_EXPORTER_EXTERNAL_URL environment variable. You will need to know the IP-address/hostname the exporter's host is reachable for you (usually something starting with 192.168. if the exporter is in your local LAN).

[sbrooke]

"FATA Error in flags: unknown flag: --external-url" is what I've been getting. I'll install the update and see if it's still happening.

[sbrooke]

Ok, got it to launch the Netatmo auth page, authorized it and:

{"error":{"code":404,"message":"method not found"}}

I'll try it again

[sbrooke]

Seems to be the same problem. I'm sure I'm just missing a step here. Appreciate all your efforts on getting this working!

[SnorreSelmer]

Old method still works for me too, no idea how long though…

tir. 6. des. 2022 kl. 01:22 skrev Robert Jacob @.***>:

Just released version 1.5.0 to push out all the non-authentication-related changes. Interestingly the old authentication still seems to work fine for me, even though it should have stopped working more than a month ago. Can anyone confirm?

For testing with the new authentication, I have updated the "new-auth" branch with the released changes. The updated Docker image is this one:

ghcr.io/xperimental/netatmo-exporter:d59fbff-new-auth

— Reply to this email directly, view it on GitHub https://github.com/xperimental/netatmo-exporter/issues/26#issuecomment-1338478716, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWMEQH2CCQ4U7T4BMTVKJTWL2BMXANCNFSM55TI7XVQ . You are receiving this because you authored the thread.Message ID: @.***>

[sbrooke]

Ok, still stuck here. I made the "new-auth" version, pulled the refresh token from Netatmo, set up the token.json file with the refresh token, and now I get a 403 Forbidden. Here's what I'm starting it with. All of the keys have been changed for sharing, of course.

./netatmo-exporter new-auth --client-id abcdefg --client-secret zyxwvu --token-file token.json

The token.json file is using this format:

{"access_token":"123456","expiry":"0001-01-01T00:00:00Z"}

What's interesting is that I didn't add the expiry stuff, the exporter must have. If I use the access token I don't get an error and I finally have stats! Using the refresh token I only get the 403 and no stats. I'll let this run for the day and see if it fails without the refresh token. Any other recommendations, provided it ends up failing on me later today?

[xperimental]

@sbrooke The NetAtmo website now issues tokens with both access_token and refresh_token, if you put that into the token.json, it should work. Try adding an expiry a few hours in the future if it doesn't work without the expiry.

Two alternatives:

• Try the "auth-tool" from the same-named branch. It will produce a proper token.json complete with valid expiry.
• If everything else fails, the old authentication still worked fine last time I checked. It still might.

Need to clean this up and write proper documentation sometime soon, but have other issues to deal with right now.

[sbrooke]

I'm waiting to make sure it updates properly, but the problem is that I had "access_token" in the file instead of "refresh_token".

Now, it seems when I run the bash script in systems it's reading the token file for some period of time and then overwriting it with a blank file.

[sbrooke]

@xperimental I created a new issue specific to the token.json.

Really appreciate you doing this! Speaking for myself, it's just for my home dashboard, so no need to rush. Thanks!

[xperimental]

@sbrooke This might be a misunderstanding on my side, but it should not work with only the refresh_token in the file, because the access_token is the one actually used for communicating with the API, while the refresh-token is used to get a new access-token when it has expired (hence the expiry-date, which is sadly missing from the one you get from the NetAtmo website).

There should probably be some error-handling in the exporter that rejects a token which has missing information, which will probably be the next thing to do. I'm also pretty sure that the "refresh-token with no expiry" case is not handled correctly yet.

[sbrooke]

That's interesting, because it definitely works the first time with only the refresh token and then updates the file with the rest of the info. I agree that it sounds like some error handling needs to be in place, but it also looks like the refresh isn't happening correctly. I'll run another test today to see if it's still happening right at 3 hours. Maybe the token refresh process isn't happening in the right window for Netatmo to be happy about it?

[tukane]

Hi, is it now possible to use the new auth method with the docker image? If yes, with variables do I have to use? I've already generated an Access and Refresh token on the Netatmo API website. Would love to get this working again.

[xperimental]

@tukane The new auth method is not merged to the default branch yet, so it is not part of the "normal Docker image". It is a breaking change and I wanted to clean it up first a bit, as the user experience is quite bad compared to the username/password authentication used before.

Working on this has been slow the last months, because I was busy with other things. And the last time I checked, the old authentication method still worked fine (it might have changed now, I did not try it again for a few months), so this change also had no "urgency".

That said, the Docker images mentioned in this issue for testing the implementation should still work properly. I'll probably get back on this topic in the near future. One alternative idea that I had was asking NetAtmo if it is really necessary to stop supporting the old authentication method, as it is much simpler for someone just monitoring their own account. But I don't know if they are open to any suggestions.

[SnorreSelmer]

I use the old method, and I still get data, so confirmed still working (at least for existing setups)

tir. 13. jun. 2023 kl. 15:09 skrev Robert Jacob @.***>:

@tukane https://github.com/tukane The new auth method is not merged to the default branch yet, so it is not part of the "normal Docker image". It is a breaking change and I wanted to clean it up first a bit, as the user experience is quite bad compared to the username/password authentication used before.

Working on this has been slow the last months, because I was busy with other things. And the last time I checked, the old authentication method still worked fine (it might have changed now, I did not try it again for a few months), so this change also had no "urgency".

That said, the Docker images mentioned in this issue for testing the implementation should still work properly. I'll probably get back on this topic in the near future. One alternative idea that I had was asking NetAtmo if it is really necessary to stop supporting the old authentication method, as it is much simpler for someone just monitoring their own account. But I don't know if they are open to any suggestions.

— Reply to this email directly, view it on GitHub https://github.com/xperimental/netatmo-exporter/issues/26#issuecomment-1589282478, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWMEQDCHZLP4A3BMHWHCXDXLBRCFANCNFSM55TI7XVQ . You are receiving this because you authored the thread.Message ID: @.***>

[tukane]

I get an error message in the docker container, which worked fine the last few month:

level=fatal msg="Error creating client: Post \"https://api.netatmo.net/oauth2/token\": dial tcp: lookup api.netatmo.net on 127.0.0.11:53: server misbehaving"

I use the old method, and I still get data, so confirmed still working (at least for existing setups) tir. 13. jun. 2023 kl. 15:09 skrev Robert Jacob @.>: … @tukane https://github.com/tukane The new auth method is not merged to the default branch yet, so it is not part of the "normal Docker image". It is a breaking change and I wanted to clean it up first a bit, as the user experience is quite bad compared to the username/password authentication used before. Working on this has been slow the last months, because I was busy with other things. And the last time I checked, the old authentication method still worked fine (it might have changed now, I did not try it again for a few months), so this change also had no "urgency". That said, the Docker images mentioned in this issue for testing the implementation should still work properly. I'll probably get back on this topic in the near future. One alternative idea that I had was asking NetAtmo if it is really necessary to stop supporting the old authentication method, as it is much simpler for someone just monitoring their own account. But I don't know if they are open to any suggestions. — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWMEQDCHZLP4A3BMHWHCXDXLBRCFANCNFSM55TI7XVQ . You are receiving this because you authored the thread.Message ID: @.>

[xperimental]

@tukane Your error has nothing to do with the discussion in this issue. The error message rather indicates that something's wrong with your local DNS server.

[sbrooke]

Started failing with a Error creating client: oauth2: cannot fetch token: 400\nResponse: {\"error\":\"unauthorized_client\"} message yesterday. Anyone else or is this just a problem on my end?

[giuliomagnifico]

Me too since few hours ago: https://github.com/xperimental/netatmo-exporter/issues/44

Maybe it's only a Netamo issue?

[xperimental]

Just checked, I'm getting the same error. :disappointed:

So it seems like Netatmo finally did the change they announced for October last year. I had hoped a bit that they see reason and keep the password authentication for accessing the sensors of the own account. The "client credentials" documentation page also still exists and and still lists October last year as deprecation date.

So, I guess this means I finally need to get the new authentication branch cleaned up, merged and released. I'm sorry for the delay, I should probably have used the past months to do this, but there have been other things that needed my attention.

But there are also good news: the code in the "new-auth" branch still works and the Docker images I produced earlier this year are still available. To save you the scrolling, here's the image name again:

ghcr.io/xperimental/netatmo-exporter:d59fbff-new-auth

The main downside of the new authentication is that the exporter is not able to start up on its own on the first run or when it has been stopped for a while (and the token expired). Once it is set up, it can run indefinitely without user interaction.

If anyone of you has an idea on how to solve the OAuth authentication in a more "admin-friendly" way, then please let me know. The one improvement that already came up in this thread (but is also not properly documented yet) is using Netatmo's website to get the first token and let the exporter immediately renew it (the token from Netatmo's website does not contain information about the expiry time).

[giuliomagnifico]

Okay thanks, but I'm building it from source and I'm also starting the service via systemd file, so what I have to do to grab the "new auth" branch and what are the new commands to use to launch the exporter? Because the readme is the same in the new branch, or I'm missing something?

At the moment I'm using:

netatmo-exporter -s xxx -i xxx -p xxx -u giuliomagnifico@gmail.com --refresh-interval 2m

Edit, I tried:

git clone --single-branch --branch new-auth https://github.com/xperimental/netatmo-exporter

and then compile and restart the exporter with the same/old commands, but I'm getting the same error/issue.

[xperimental]

@giuliomagnifico Updating the documentation is one of the things that is missing and why it's not merged yet. Also there's a few small bugs.

The --username and --password parameters do not exist anymore in that version.

You need to pass it the client-id, client-secret and a token-file (so the token can be saved). Once the exporter is running, visit /authorize in a browser (for example http://localhost:9210/authorize) and it should take you through the OAuth process. Once that is complete the metrics should show up again.

When you exit the exporter, the token will be saved to the token-file, so when it is restarted while the token is still valid you do not need to authorize again.

[giuliomagnifico]

@xperimental got it thanks but I'm having issues.

When I start the new-auth exporter to get the token I have to specify my Prometheus server url, so I launch it with:

./netatmo-exporter -s xxx -i xxx --external-url 192.168.1.6

it opens the Netatmo page correctly

but when I accept it returns me this error:

{"error":{"code":404,"message":"method not found"}}

it should me return the token file? Or what else?

And on the CLI i get this error:

INFO Listen on :9210...
ERRO Error during refresh: no token available

Thanks again and sorry for the questions!

[xperimental]

@giuliomagnifico Can you try with setting --external-url to a URL, so, in your case, probably something like http://192.168.1.6:9210, the callback path will be appended automatically.

The URL just needs to be valid&reachable for you, so if you are able to access the exporter as 127.0.0.1:9210, then you can leave it at the default.

[xperimental]

Small update: I have updated the new-auth branch with some small fixes and a small UI improvement (I hope :slightly_smiling_face: ). And the Docker image has been updated with the latest changes:

ghcr.io/xperimental/netatmo-exporter:0dccd27-new-auth

This should also fix #37 and provides some metrics about the token itself, providing some insight into refreshing tokens and validity.

That's it for now :sleeping_bed:

[giuliomagnifico]

@xperimental okay found the issue, I have to specify the full url with also the port --external-url http://192.168.1.6:9210 thanks for the support.

Now the browser says Authenticated. but it doesn't show or download the token.json or anything else, in the Netatmo-exporter folder I also don't find the token file. It's my issue?

(I'm using the new exporter updated from your latest comment/commits)

Thanks again for this exporter and your work!