No support for username/password from October ‘22

[SnorreSelmer]

I got this email yesterday:

To improve the security of our products, we inform you that the Client Credentials grant type method will be completely removed. It will no longer be possible to authenticate with the username and password of the user. 

The effective date of this update is October 2022.

How can you authenticate with Netatmo API ?
From this date, the OAuth2 authorization code flow must be followed for authentication.

Is this exporter ready for this change? What needs to be altered in the compose-file?

[giuliomagnifico]

I reply to myself just to say that now I got it working but only by generating manually/via web interface the token,

then I created the token.json file and it works

{"access_token":"1234","refresh_token":"1234"}

...without the expiration date. I'll try to see what will happend despite the warning WARN Restored token has no expiry time! Expiry set in 30 Minutes.

Theorically there's also the refresh token so it should auto-reload it?

@xperimental okay found the issue, I have to specify the full url with also the port --external-url http://192.168.1.6:9210 thanks for the support.

Now the browser says Authenticated. but it doesn't show or download the token.json or anything else, in the Netatmo-exporter folder I also don't find the token file. It's my issue?

(I'm using the new exporter updated from your latest comment/commits)

Thanks again for this exporter and your work!

[SnorreSelmer]

I’m just chiming in here to say that I’m spun up some services at work where the official docs said “spin up container X first, run command Y against it, then spin up the rest of the compose-file.” If we can have that as the workflow to get this working, then that’s perfectly fine to me. I guess we’ll need to attach some local storage (./token) for the token as well?

fre. 14. jul. 2023 kl. 08:13 skrev giuliomagnifico @.***

:

I reply to myself just to say that now I got it working but only by generating manually/via web interface the token, then I created the token.json file and it works

{"access_token":"1234","refresh_token":"1234"}

...without the expiration date. I'll try to see what will happend despite the warning WARN Restored token has no expiry time! Expiry set in 30 Minutes.

Theorically there's also the refresh token so it should auto-reload it?

@xperimental https://github.com/xperimental okay found the issue, I have to specify the full url with also the port --external-url http://192.168.1.6:9210 thanks for the support.

Now the browser says Authenticated. but it doesn't show or download the token.json or anything else, in the Netatmo-exporter folder I also don't find the token file. It's my issue?

(I'm using the new exporter updated from your latest https://github.com/xperimental/netatmo-exporter/issues/26#issuecomment-1635107423 comment/commits)

Thanks again for this exporter and your work!

— Reply to this email directly, view it on GitHub https://github.com/xperimental/netatmo-exporter/issues/26#issuecomment-1635331800, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWMEQHRPJLDDCDH5KF6IIDXQDPR5ANCNFSM55TI7XVQ . You are receiving this because you authored the thread.Message ID: @.***>

[xperimental]

@giuliomagnifico

Now the browser says Authenticated. but it doesn't show or download the token.json or anything else, in the Netatmo-exporter folder I also don't find the token file. It's my issue?

The token.json is currently only touched twice: once on startup to load the token and once on shutdown to write it out again. When there was no token.json on startup and you authenticate using the web interface in the exporter the file will not show up until the exporter is stopped.

I made a basic interface in the exporter yesterday at /, which shows the current token state and expiry.

...without the expiration date. I'll try to see what will happend despite the warning WARN Restored token has no expiry time! Expiry set in 30 Minutes.

Theorically there's also the refresh token so it should auto-reload it?

That's something I added yesterday as well. It will warn about missing bits in the token. When the expiry is missing but there's a refresh token it will set a short expiry (I picked 30min) which will cause it to refresh the token after that time expires. I also found out that it works with only the refresh token, it will then refresh to a proper token immediately. Maybe that's the easiest solution for the future.

---

@SnorreSelmer

I’m just chiming in here to say that I’m spun up some services at work where the official docs said “spin up container X first, run command Y against it, then spin up the rest of the compose-file.” If we can have that as the workflow to get this working, then that’s perfectly fine to me.

Thanks for the feedback.

I guess we’ll need to attach some local storage (./token) for the token as well?

Yes, if you want to persist the token across restarts it now needs some kind of storage. I made my exporter a StateFulSet with one replica.

---

btw: one quirk in the exporter that is because on how it's caching the NetAtmo data internally is that after authentication is successful, the metrics will only show up on the second scrape. This has been the case with the previous password authentication as well, but might catch people off-guard when now looking at "does the authentication work properly".

[tom-43]

Hi there! I have the same problems as all of you, but I continue to get errors after your solutions. I use the netatmo-exporter with Docker and I'm not quite sure how to include the external-url in the ENVs. If I simply create external-url there and specify my IP I am still redirected to 127.0.0.1. I have also tried to create the token.json manually, but the file is not even considered by the exporter.

BTW: Thanks for your great work @xperimental !

[tom-43]

I got it working! I simply did not use ENV variables now, but used CMD variables.

My only problem is that I now have to re-authenticate every time I reboot (which fortunately is not often). No data is stored in token.json for me. Is there any way to solve this?

[xperimental]

@tschmitt1605

I got it working! I simply did not use ENV variables now, but used CMD variables.

The configuration options can also be set with environment variables. They do look a bit different than the command-line arguments, though. Both are listed here.

No data is stored in token.json for me. Is there any way to solve this?

You mention Docker. Do write the file to a location which is a volume? Otherwise it will not persist across recreations of the container. It should still persist across restarts of the container though.

[sbrooke]

Pulled the new-auth branch again and created the simple token.json referenced by guiliomagnifico. Restarted the service and it's all good for now. I'll let you know if it quits after a period.

Thanks for jumping on this quickly! My wife also thanks you for getting her weather stats back! :)

[giuliomagnifico]

@xperimental @sbrooke I can confirm that manually adding the access and refresh tokens to the token.json is working (after 12h) and after some reboots.

root@Grafana:~# service netatmo-exporter status
● netatmo-exporter.service - netatmo-exporter
     Loaded: loaded (/etc/systemd/system/netatmo-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-07-14 23:00:18 CEST; 10h ago
   Main PID: 403 (netatmo-exporte)
      Tasks: 10 (limit: 4531)
        CPU: 16.593s
     CGroup: /system.slice/netatmo-exporter.service
             └─403 /root/netatmo-exporter/netatmo-exporter -s 1234 -i 1234 --to
ken-file token.json

Jul 14 23:00:18 Grafana systemd[1]: Started netatmo-exporter.
Jul 14 23:00:18 Grafana netatmo-exporter[403]: level=info msg="Listen on :9210..."

I took advantage of the change in Netatmo's autenthication change to clean up my Prometheus and Grafana server, I deleted old data and messed data, removed the Smart Thermostat sections (because the Netatmo-energy-exporter is not updated for the recent changes). Now it seems that we can go ahead with this setup, hope there will not be other weird changes in API from Netatmo.

[tom-43]

@xperimental

The configuration options can also be set with environment variables. They do look a bit different than the command-line arguments, though. Both are listed here.

Ok, thanks for the link.

You mention Docker. Do write the file to a location which is a volume? Otherwise it will not persist across recreations of the container. It should still persist across restarts of the container though.

I have a bind from /docker/netatmo-exporter/token.json to /token.json . So it should work normally.

Meanwhile, I have another error that occurred overnight. I get the following error: Error getting token: Post "https://api.netatmo.net/oauth2/token": context canceled.

I hope you can help me.

[xperimental]

@sbrooke @giuliomagnifico Reading this thread I come to the conclusion that the easiest route for adding the token is probably getting it from NetAtmo's website and entering it into the exporter's configuration manually, so I'll try to make that route simpler (the token.json still feels a bit cumbersome, even if it works).

So far, I had no issues with using /authorize of the exporter to go through the OAuth process and I wonder if I'm missing something that breaks it for you, because I know how it should work. Would you like to try out the web-flow again to either confirm that it works or tell me where it breaks for you, now that you have a working configuration?

---

@tschmitt1605

Meanwhile, I have another error that occurred overnight. I get the following error: Error getting token: Post "https://api.netatmo.net/oauth2/token": context canceled.

This sounds like a timeout. Did it only happen once? Did the exporter recover from it?

---

In case you missed it: the new_auth branch also adds new metrics for checking on the token:

• netatmo_exporter_token_valid, which is 1 when the exporter has a valid token
• netatmo_exporter_token_expiry_time, which contains the timestamp when the token will expire. netatmo_exporter_token_expiry_time - time() should yield a sawtooth curve with a peak at ~3h (the tokens seem to currently have a lifetime of 3h), good for monitoring the refresh of the token

If everything goes as planned, I should have the 2.0.0 out tomorrow. I probably don't have time for coding today.

[giuliomagnifico]

@xperimental

Would you like to try out the web-flow again to either confirm that it works or tell me where it breaks for you, now that you have a working configuration?

For me the process of OAuth works but it doens't download the token file or nothing else, I see only an Authorized word after I press "Yes I accept" on the Netamo website. This is why I went to manually ad the data to the token.json file.

If everything goes as planned, I should have the 2.0.0 out tomorrow. I probably don't have time for coding today.

Great thanks for it! 🥳

[tom-43]

This sounds like a timeout. Did it only happen once? Did the exporter recover from it?

It happened only once, but I can't give you more data about it, because I restarted the container immediately and included the new ENV varibals. Now retrieving the data from token.json works as well.

For me the process of OAuth works but it doens't download the token file or nothing else, I see only an Authorized word after I press "Yes I accept" on the Netamo website. This is why I went to manually ad the data to the token.json file.

I have the same problem and I have also entered the dates manually.

If everything goes as planned, I should have the 2.0.0 out tomorrow. I probably don't have time for coding today.

Nice! Thank you!

[sbrooke]

Sad to report it zeroed out the token.json file on me only a little while after it started working. When I manually create the token.json the process has a message about "no expiry, setting it to 30 minutes".

I just tried the OAuth and it worked, and updated the token.json file. It also added an expiry date to it. However, when I was going through the process it tried to do a callback to 127.0.0.1 and failed. I'm running this on a Ubuntu VM and did the OAuth from my laptop. However, it's not actually pulling stats. I'm getting the following:

Jul 15 12:32:45 netatmo systemd[1]: Starting Netatmo... Jul 15 12:32:45 netatmo bash[994]: level=info msg="Listen on :9210..." Jul 15 12:32:53 netatmo bash[994]: level=error msg="Error during refresh: bad HTTP return code: 403"

Close to 30 minutes and it zeroed the file again.

[xperimental]

@sbrooke

Close to 30 minutes and it zeroed the file again.

I'm confused by this. The exporter only touches the file on startup and shutdown, not while it is running. Is something else (not the exporter itself) causing the exporter to restart or overwriting the file?

[xperimental]

@giuliomagnifico

For me the process of OAuth works but it doens't download the token file or nothing else, I see only an Authorized word after I press "Yes I accept" on the Netamo website.

The token file is not written until the exporter exits. If you see the callback (the page with "Authorized" on it) then the file will not immediately appear. The exporter should have the token in memory though and be able to scrape metrics.

[sbrooke]

@sbrooke

Close to 30 minutes and it zeroed the file again.

I'm confused by this. The exporter only touches the file on startup and shutdown, not while it is running. Is something else (not the exporter itself) causing the exporter to restart or overwriting the file?

I'm running it as a service. I'm not using Docker. Don't know if that has anything to do with it. Also, I don't know if it's missing a step when I try to do the authentication through the exporter's web interface. In other words, I click the link you set up in the root web page, it takes me to Netatmo to auth, I complete that and it returns me to 127.0.0.1, which is wrong because I'm doing this from my PC and not from the Exporter server. I get an error message there, but it does update the token.json to include the updated info.

[sbrooke]

I can also see at the root web page that:

`You have a token.

Token is valid until 2023-07-16 14:55:30.281143226 +0000 UTC (27m15.683161881s)`

[sbrooke]

Sorry for the multiple posts. I just copied the old (non-expiry) token.json in and rebooted and I have the above message, but it's not connecting to Netatmo successfully. The Exporter has a stats page but none of the Netatmo stats are listed and the service is showing a 403 error message.

I'm going to try removing the token.json and going through the web page auth process again.

[sbrooke]

It's quite possible my service is formatted incorrectly:

/home/netatmo/netatmo-exporter/netatmo-exporter --client-id BLAH --client-secret BLEH --token-file /home/netatmo/netatmo-exporter/token.json

[xperimental]

@sbrooke

In other words, I click the link you set up in the root web page, it takes me to Netatmo to auth, I complete that and it returns me to 127.0.0.1, which is wrong because I'm doing this from my PC and not from the Exporter server.

If you're not accessing the /authorize URL from 127.0.0.1 then you need to update the --external-url in the exporter, so that it generates the correct URL for NetAtmo to redirect you to. The "flow" is something like this:

• you access the exporter at /authorize, it will redirect you to the NetAtmo site
• you click "Yes, I accept" on NetAtmo's website
• it will try to redirect you back to the exporter (to a /callback URL). This is the important step, as this will enable the exporter to get the token. If it was successful, you should see a "Authorized" message
• the exporter now has the token and should be able to scrape the metrics (it will not write the token-file at this point)
• when the exporter shuts down, it will write its current token to the token-file

I just copied the old (non-expiry) token.json in and rebooted and I have the above message, but it's not connecting to Netatmo successfully.

The tokens issues by NetAtmo are only valid for ~3h, this is also true for the ones generated using their website. This means that you can not "reuse" an old token, even if you created it manually using the website. When the token has no expiry, the exporter will "guess" a lifetime of 30min, to renew the token earlier. When the exporter has a token with a refresh-token, it will automatically renew the token when it expires.

[xperimental]

@sbrooke Two questions that just occurred to me:

1) What "scope" do you select when creating the token on the website? (read_station is the one the exporter currently needs) 2) What information do you put in the token.json file when you create it manually? Just the access-token or also the refresh-token?

---

I'm running it as a service. I'm not using Docker. Don't know if that has anything to do with it.

No, it should also work without using Docker. In fact that's how I'm testing locally as well. It should make it simpler to configure instead of harder, as you don't need to worry about the container namespacing.

---

I'm currently adding a way to "simply" get the token manually created on the website into the exporter.

[sbrooke]

Ok, ran it directly with --external-url http://192.168.1.19:9210 and the authentication succeeded this time. And I think it took a restart but I'm getting weather stats now. Token shows ~3 hours. Running the service without the --external-url appears to be working, although it's not certain yet.

Crossing my fingers...I'll let you know in a couple of hours.

[xperimental]

Running the service without the --external-url appears to be working, although it's not certain yet.

The external-url is only needed for the last part of the authentication (the callback), so once the exporter has a refreshable token you should be fine (it is not needed for the refresh).

[xperimental]

A few new changes have been incorporated into new_auth and I have built a new Docker image to test with:

ghcr.io/xperimental/netatmo-exporter:db281b5-new-auth

The main change since the last build is that I have extended the "homepage". There's a textbox for filling in the refresh_token from the NetAtmo developer console now, which should make setup easier. I'd be thankful if someone would be able to test this and give me some feedback (please don't provide any feedback on how the page looks, I know it is awful :wink: ).

I have also extended the info and warning messages for loading/saving the token file a bit. The Docker image has been extended to provide a volume for the token-file by default. This can be customized, of course, but might make it easier to setup.

---

If the new version works for most of you and there are no showstoppers anymore then I think this might be the version that will get the initial 2.0 release. :slightly_smiling_face:

[xperimental]

There's also a new document explaining how to authenticate. Feedback appreciated. :slightly_smiling_face:

[theonlydoo]

There's also a new document explaining how to authenticate. Feedback appreciated. 🙂

Tested and validated! works perfectly, thanks!

[tom-43]

Everything works for me as well. I think you can publish the current new auth as main now.

Thanks for your commitment and the exporter.

[sbrooke]

Grabbed the latest main this morning and it's working fine. If I run it manually. Doesn't like to run as a service. Probably a config problem on my end, so I'll sort that out.

[tom-43]

Grabbed the latest main this morning and it's working fine. If I run it manually. Doesn't like to run as a service. Probably a config problem on my end, so I'll sort that out.

I will try it later if I have the same problems when I run it as a service?

@xperimental

Do you need help with the docs?

[xperimental]

@sbrooke

Doesn't like to run as a service.

Do you get any error? Based on your previous messages, I'm assuming systemd service unit... do you use a user that has read/write access to the path you use as a token-file? Any of the sandboxing features of systemd active?

@tschmitt1605

Do you need help with the docs?

My hope is that the updated readme and the new files in doc/ are sufficient. :slightly_smiling_face:

I think you can publish the current new auth as main now.

I have already merged the PR on Sunday, meaning the new version is already available in ghcr.io/xperimental/netatmo-exporter:master.

The 2.0.0 release might happen today. That will update latest.

---

One thought I had while working on this issue is whether it would make sense to add some "template configuration files" in a contrib/ folder... I'm using a StatefulSet, so I can provide this. There seem to be a few people using systemd to run it as a normal service (which I approve of as well :+1: :wink: ) ...

There has also always been the question of "should I build binaries as well in addition to the images"... I think that might be something to improve post-2.0.0 as well.

[tom-43]

One thought I had while working on this issue is whether it would make sense to add some "template configuration files" in a contrib/ folder...

I think this is a very good idea. Especially for people that are not so interested in programming…

[sbrooke]

Do you get any error? Based on your previous messages, I'm assuming systemd service unit... do you use a user that has read/write access to the path you use as a token-file? Any of the sandboxing features of systemd active?

I'll go back and look at the permissions. What's odd is that it's touching the token.json file, but then blanking it. In other words, the token is present in the file until something causes it to be overwritten with an empty file. Doesn't seem to be an issue with the ability to write the file, just that it doesn't have a token to write into it.

Here's the service:

`[Unit] After=network.target Description=Netatmo

[Service] Type=forking User=netatmo WorkingDirectory=/usr/bin ExecStart=/usr/bin/bash -c "/usr/bin/netatmo.sh" Restart=always

[Install] WantedBy=multi-user.target`

Probably something wrong in what I have there.

[sbrooke]

The netatmo.sh file is simply running the command string I used to launch it. Perhaps the service isn't seeing it as a running program and restarting it constantly? New instance wipes out the token.json to start over again?

[xperimental]

@sbrooke If you like, we can debug your case, but I think it would be nice to have a separate issue for it. Would you like to create one?

Why do you need the shell script and Type=forking? Only thing I can think of right now is setting up the environment and I'd think using Environment or EnvironmentFile directives is better suited to that.

And when you say the token-file is "blanked" is it really completely empty (0b) or is it just a JSON with empty values?

[xperimental]

netatmo-exporter 2.0.0 has just been released :rocket: Thank you all for your patience!

[sbrooke]

@sbrooke If you like, we can debug your case, but I think it would be nice to have a separate issue for it. Would you like to create one?

Why do you need the shell script and Type=forking? Only thing I can think of right now is setting up the environment and I'd think using Environment or EnvironmentFile directives is better suited to that.

And when you say the token-file is "blanked" is it really completely empty (0b) or is it just a JSON with empty values?

Totally agree, this issue is closed. Thank you!

[m-arx]

I cannot get it running as a docker container. I have build an image with 2.0.0

I have created an env file with this content:

NETATMO_CLIENT_ID='lesecretsauce'
NETATMO_CLIENT_PASSWORD='cannotreadme'
NETATMO_CLIENT_SECRET='gibberish'
NETATMO_CLIENT_USERNAME='me@world.com'
NETATMO_EXPORTER_EXTERNAL_URL='http://192.168.2.40:9210'

I have tried the same without the external url, tunnelling the 9210 port from my docker machine to my notebook and calling it from 127.0.0.1, I always end up with "Error processing code: user did not accept" after clicking on "authorize".

also I have tried creating a token.json with a start token from the development.netatmo.com site (and mounting it into the docker container and creating an env for it).

[tom-43]

@m-arx You already have too many ENV Varibals. With the new authentication, username and password are no longer needed. The following ENVs are sufficient:

NETATMO_CLIENT_ID='lesecretsauce'
NETATMO_CLIENT_SECRET='gibberish'
NETATMO_EXPORTER_EXTERNAL_URL='http://192.168.2.40:9210'

If you want to save your token.json file separately you also have to add the file path: NETATMO_EXPORTER_TOKEN_FILE=/token.json But then you have to include the token.json from the system as well.

If you have any questions or it still doesn't work, please contact me again.

[m-arx]

If you have any questions or it still doesn't work, please contact me again.

Thank you, I have now changed my env file according to the above, but when clicking on "authorizing here"I get a 404 from netatmo when calling the url "https://api.netatmo.net/oauth2/'http://192.168.2.40:9210'/auth/callback?error=invalid_client&state=definitelyrandom"

The return in the browser is "{"error":{"code":404,"message":"method not found"}}"

my API Client ID/secret was working for years, so I thought maybe something changed and I need to create a new one - did it with the same outcome.

[tom-43]

The return in the browser is "{"error":{"code":404,"message":"method not found"}}"

If you get this error, you may have set the wrong external-url. I had taken the one from your example. Check the iP again and then everything should work.

[xperimental]

@m-arx Didn't see it yesterday, but this has happened previously in another question: The values specified in the environment file are forwarded as-is to the exporter. Do not put quotes (' or ") around the values.

If this needs further debugging, can you please create a new issue to discuss this in?

[giuliomagnifico]

Sorry for the delay @xperimental, I had the time to test the new branch v2.0 compiled - not docker and not the experimental one with the token- only now (that I'm not at work), and I can confirm that it works fine always using/filling manually the token.json file.

What I want to suggest is to have the execute permission on the token.json file, like a 755, because without it I'm not sure that the exporter can read it (I had some issues).

root@Grafana:~/netatmo-exporter# service netatmo-exporter status
● netatmo-exporter.service - netatmo-exporter
     Loaded: loaded (/etc/systemd/system/netatmo-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-07-22 09:29:56 CEST; 34s ago
   Main PID: 127267 (netatmo-exporte)
      Tasks: 9 (limit: 4531)
        CPU: 161ms
     CGroup: /system.slice/netatmo-exporter.service
             └─127267 /root/netatmo-exporter/netatmo-exporter -s 123 -i 123 -
-token-file token.json

Jul 22 09:29:56 Grafana systemd[1]: Started netatmo-exporter.
Jul 22 09:29:56 Grafana netatmo-exporter[127267]: level=info msg="Loaded token from token.json."
Jul 22 09:29:56 Grafana netatmo-exporter[127267]: level=info msg="Listen on :9210..."

[xperimental]

@giuliomagnifico Good to hear that it works for you.

What I want to suggest is to have the execute permission on the token.json file

While the exporter needs both read and write access to the token file, it does not need the execute bit to be set. If your current configuration works with the file having a mode of 0755 (-rwxr-xr-x) it should work just the same with 0644 (-rw-r--r--). I don't know what your service unit file looks like, but I would recommend running the exporter as a non-root user, putting the files outside of /root and setting the configuration using environment variables. I have added an example unit file a few days ago, which you can use as a starting-point.

We can talk more about this, but please open a separate issue for that. :slightly_smiling_face:

[SnorreSelmer]

Finally got my server back up after moving house, updated the container to the latest version, banged my head against the wall because I didn't read the instructions properly (NETATMO_EXPORTER_EXTERNAL_URL needs both "http://" and port-number, duh), but the setup works great and I have data flowing in just fine! Two thumbs up!