authentication failure

[peemot]

The script stopped working a day or two ago.

[ERROR] failed with exception: authentication failure: did you enter valid credentials?

[zxxxh]

Confirmed - I have receive the same message. Debug shows additional messages:

Access denied This website is using a security service to protect itself from online attacks.

[y3kde]

Same here: [ERROR] failed with exception: authentication failure: did you enter valid credentials?

[random1781]

Garmin seems to be blocking requests based on the provided User-Agent. I now get an authentication failure error with the following debug message:

  <p>This website is using a security service to protect itself from online attacks.</p>
  <ul class="cferror_details">
    <li>Ray ID: rayid</li>
    <li>Timestamp: 2021-05-07 13:11:38 UTC</li>
    <li>Your IP address: #.#.#.#</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: sso.garmin.com/sso/signin?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&amp;gauthHost=https%3A%2F%2Fsso.garmin.com%2Fsso </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_16F223</li>
    <li>User-Agent: python-requests/2.25.1</li>
  </ul>

If I add a defined User-Agent of a browser to the request, the authentication succeeds and my activities are downloaded as expected.

I'm not sure if this is a temporary or long-term problem, but I added fake-useragent to generate a User-Agent for long term.

[pavlinux]

diff --git a/garminexport/garminclient.py b/garminexport/garminclient.py
index 2ac4674..45ef6aa 100755
--- a/garminexport/garminclient.py
+++ b/garminexport/garminclient.py
@@ -118,7 +118,8 @@ class GarminClient(object):
             "embed": "false",
             "_csrf": self._get_csrf_token(),
         }
-        headers = {'origin': 'https://sso.garmin.com'}
+        headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.134 Safari/537.',
+                   'origin': 'https://sso.garmin.com'}
         auth_response = self.session.post(
             SSO_SIGNIN_URL, headers=headers, params=self._auth_params(), data=form_data)
         log.debug("got auth response: %s", auth_response.text)

user-agent.patch.zip

[pavlinux]

Random User Agent

diff --git a/garminexport/garminclient.py b/garminexport/garminclient.py
index 2ac4674..70ffe67 100755
--- a/garminexport/garminclient.py
+++ b/garminexport/garminclient.py
@@ -14,6 +14,7 @@ from datetime import timedelta, datetime
 from builtins import range
 from functools import wraps
 from io import BytesIO
+import random

 import dateutil
 import dateutil.parser
@@ -112,13 +113,22 @@ class GarminClient(object):
     def _authenticate(self):
         log.info("authenticating user ...")

+        with open('user-agents.txt', 'r') as f:
+             random.seed()
+             fakeua = random.choice(f.readlines()).replace('\n', '')
+             f.close()
+
         form_data = {
             "username": self.username,
             "password": self.password,
             "embed": "false",
             "_csrf": self._get_csrf_token(),
         }
-        headers = {'origin': 'https://sso.garmin.com'}
+
+        log.debug("User-Agent: '%s'", fakeua)
+        headers = {'User-Agent': fakeua,
+                   'origin': 'https://sso.garmin.com'}
+
         auth_response = self.session.post(
             SSO_SIGNIN_URL, headers=headers, params=self._auth_params(), data=form_data)
         log.debug("got auth response: %s", auth_response.text)

Put file user-agents.txt into call script directory.

user-agents.zip

[pavlinux]

Does not love an Opera )))

2021-05-08 13:32:44,555 [INFO] backing up formats: fit 2021-05-08 13:32:44,555 [INFO] authenticating user ... 2021-05-08 13:32:44,559 [INFO] fetching CSRF token ... 2021-05-08 13:32:44,885 [INFO] User-Agent: 'Opera/9.80 (Windows NT 6.1; WOW64; U; pt) Presto/2.10.229 Version/11.62' 2021-05-08 13:32:45,494 [ERROR] failed with exception: authentication failure: did you enter valid credentials?

[micky-gee]

The solution in this pull request https://github.com/petergardfjall/garminexport/pull/77/files works really well.

Nice fake useragent.

[petergardfjall]

I've addressed this issue in https://github.com/petergardfjall/garminexport/commit/33b83548675823f8f872a491363dffc9ba091992.

It allows a custom User-Agent to be supplied. Either via command-line.

garmin-backup --user-agent='Mozilla/5.0 (X11; Ubuntu; Linux i686 on x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2811.59 Safari/537.36'

If you prefer to rotate user agent, use a tool such as user_agent:

# installs the ua cli tool
pip install --user user_agent

garmin-backup --user-agent=$(ua)

For library clients, one can pass a user_agent_fn function that, when called, produces a User-Agent string to be used as User-Agent for the remainder of the session.

[petergardfjall]

Released 0.4.0 on pypi.org

[pavlinux]

--user-agent=$(ua)'

How this work from bash/sh ?

[petergardfjall]

@pavlinux how do you mean? That is bash syntax.

[micky-gee]

The problem is, that without even additional command line flags, it won't work. I think a better philosophy to be: command line tool should work with as little flags as possible, the bare minimum being credentials.

So without adding this: garmin-backup --user-agent=$(ua)

It fails.

The other point to think about: this tool has gained sufficient popularity for Garmin to block it. Which means we're in the game of whack-a-mole with their system administrators. Potentially triggered by someone hitting the requests too hard? Or maybe them looking to enforce the API interface harder. If its the former, some good-behaviour rate limiting would also be in order.

Either way, if we use a standard UA, it'll get blocked again.

Also, hi :wave: Garmin. I know the chaps in Kansas are reading these comments too. We <3 you.

[MeisterP]

So without adding this: garmin-backup --user-agent=$(ua)

It fails.

That's not true. It's working without --user-agent parameter. Have you looked at commit https://github.com/petergardfjall/garminexport/commit/33b83548675823f8f872a491363dffc9ba091992?

https://github.com/petergardfjall/garminexport/blob/master/garminexport/cli/backup.py#L21

DEFAULT_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
"""The default `User-Agent` to use for HTTP requests when none is supplied by
the user.
"""

[micky-gee]

Sorry, I made my point poorly.

The problem is, that without even additional command line flags, it won't work. Read as: without the additional command line flags it likely won't continue to work.

How long before Garmin blocks that default user agent? Remember, again, Garmin saw enough requests from this tool to block it; this is whack-a-mole territory.

[petergardfjall]

The other point to think about: this tool has gained sufficient popularity for Garmin to block it. Which means we're in the game of whack-a-mole with their system administrators. Potentially triggered by someone hitting the requests too hard? Or maybe them looking to enforce the API interface harder. If its the former, some good-behaviour rate limiting would also be in order.

Point taken. Although the idea is flattering in some sense, I mean this tool having gained enough popularity for Garmin to actively try to block it, I'm not sure about that conclusion (maybe I'm too naive).

Garmin is definitely working on tightening up their API access to try and filter out uninvited guests like this one (I don't know how many times I've rewritten the authentication code to go through yet another weird hoop...), but I'm not sure if they specifically have tried to block this tool, but generally anything that doesn't look like a web-browser (before this change, the User-Agent was set to requests/2.25 or something to that effect, which is a simple give-away). I don't think they can blacklist a popular user-agent like the default one, although they might be using smarter discovery filters that actually recognize the "fingerprints" of that browser (headers to be expected, etc). Obviously they aren't on that level of sophistication (yet), since things are working with the default.

We'll see how things play out. I'll keep your words in mind though. Thanks for commenting.

[random1781]

I'm not sure if they specifically have tried to block this tool

I agree with this. I think this error is actually a function of Cloudflare's filtering and is probably a common sense thing to enable. I would like to think that they're more focused on stopping another multi-million dollar ransom payout than a tool like this.

I script my syncs, so adding a stand-alone UA generator is simple.

[pavlinux]

@pavlinux how do you mean? That is bash syntax.

Sorry, I didn't know that after 'pip install --user user_agent', ua it's a utility from $PATH ))

To be honest, I hate Python :rofl:

[pavlinux]

2021-05-17 20:16:07,948 [ERROR] failed with exception: authentication failure: did you enter valid credentials?

[micky-gee]

Looks like its been blocked again… failing out on all number of UAs.

[zxxxh]

I don't think they can blacklist a popular user-agent like the default one, although they might be using smarter discovery filters that actually recognize the "fingerprints" of that browser (headers to be expected, etc). Obviously they aren't on that level of sophistication (yet), …

It looks like they did get to that point.

I wish Garmin's engineers would work on improving their products instead of preventing customers from backing up their own data.

I'm curious what the fix will end up being. Currently, I cannot authenticate with any UA, including the 'valid' one my browser supplies.

[pavlinux]

I'm curious what the fix will end up being.

https://github.com/petergardfjall/garminexport/issues/79#issuecomment-843658546