Broke again? GarminException: Did not find the display name in the profile page.

[app4g]

Seems to be able to get the Ticket, but after that, it's not getting anything.

➜ garmin-connect-export-master python gcexport.py --count 1
Welcome to Garmin Connect Exporter! [WARNING] Output directory ./2023-09-26_garmin_connect_export already exists. Will skip already-downloaded files and append to the CSV file. Username: username@gmail.com Password: Connecting to Garmin Connect... Done. Requesting Login ticket... Done. Ticket=ST-0426532-gzLYoa1LJbUhxVun3keA-cas Authenticating... Done. Getting display name...Traceback (most recent call last): File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 1349, in main(sys.argv) File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 1301, in main userstats_json = fetch_userstats(args) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 952, in fetch_userstats display_name = extract_display_name(profile_page) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 977, in extract_display_name raise GarminException('Did not find the display name in the profile page.') GarminException: Did not find the display name in the profile page.

[sgowtham]

Good morning.

I have been observing the same issue as well from about 6:30 pm Eastern Time in US. Tried it from a few different machines (2 Linux and 1 Mac) - all running the required version (or better) of Python3 to ensure it wasn’t limited to one machine’s OS/software.

I didn’t report right away thinking Garmin might be doing something on their end. I’m willing to help out any way I can to find a resolution or test the solution.

My best, G

— Gowtham, PhD Director of Research Computing, IT Res. Assoc. Professor, College of Computing Michigan Technological University

(906) 487-4096 https://sgowtham.com

On Tue, Sep 26, 2023 at 5:11 AM app4g @.***> wrote:

Seems to be able to get the Ticket, but after that, it's not getting anything.

➜ garmin-connect-export-master python gcexport.py --count 1 Welcome to Garmin Connect Exporter! [WARNING] Output directory ./2023-09-26_garmin_connect_export already exists. Will skip already-downloaded files and append to the CSV file. Username: @.*** Password: Connecting to Garmin Connect... Done. Requesting Login ticket... Done. Ticket=ST-0426532-gzLYoa1LJbUhxVun3keA-cas Authenticating... Done. Getting display name...Traceback (most recent call last): File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 1349, in main(sys.argv) File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 1301, in main userstats_json = fetch_userstats(args) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 952, in fetch_userstats display_name = extract_display_name(profile_page) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 977, in extract_display_name raise GarminException('Did not find the display name in the profile page.') GarminException: Did not find the display name in the profile page.

— Reply to this email directly, view it on GitHub https://github.com/pe-st/garmin-connect-export/issues/95, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXCPKEHWLQ62L3X456RTVLX4KL5PANCNFSM6AAAAAA5HL5DYE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[app4g]

I've tried a few different packages but those that I can get to work (as in compile in my machine) are not able to get anything from Garmin Servers as well. I was inspecting the login headers via chrome developer mode and I don't seem to see any "nk":"NT" headers which I believe I saw before (too bad I didn't take a screenshot of it previously to compare it to current)

[SimonBaars]

Hi, I started investigating this, and it appears they moved the call to get the displayName to a separate JSON call on the member page. The displayName field embedded in the page now contains an id, which is the id that call be used to do a separate JSON call to get the name. This is the precise call that now gets the name (as fullName field):

curl 'https://connect.garmin.com/connection-service/connection/connections/pagination/[id]?start=1&limit=24&displayMutedStatus=false&_=[number]' \
  -H 'authority: connect.garmin.com' \
  -H 'accept: application/json, text/javascript, */*; q=0.01' \
  -H 'accept-language: en-US,en;q=0.9,nl-NL;q=0.8,nl;q=0.7' \
  -H 'authorization: Bearer [token]' \
  -H 'baggage: sentry-environment=prod,sentry-release=connect%404.71.149,sentry-public_key=[number],sentry-trace_id=[number],sentry-sample_rate=1' \
  -H 'cookie: __cflb=[number]; _cfuvid=[number]; GarminUserPrefs=en-US; TAsessionID=[number]|NEW; notice_behavior=implied,eu; __cfruid=[number]; GARMIN-SSO=1; GARMIN-SSO-CUST-GUID=[number]; SESSIONID=[number]; JWT_FGP=[number]; SameSite=None; ADRUM_BTa=[number]; ADRUM_BT1=[number]' \
  -H 'di-backend: connectapi.garmin.com' \
  -H 'nk: NT' \
  -H 'referer: https://connect.garmin.com/modern/profile' \
  -H 'sec-ch-ua: "Google Chrome";v="117", "Not;A=Brand";v="8", "Chromium";v="117"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sentry-trace: [number]' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36' \
  -H 'x-app-ver: 4.71.1.3' \
  -H 'x-lang: en-US' \
  -H 'x-requested-with: XMLHttpRequest' \
  --compressed

Since the Garmin website uses Cloudflare, we'll have to obtain a Cloudflare token to script this request as a replacement of the current member page request.

I'm working on this.

[app4g]

Wow.. how did you managed to locate all these items? I am only using Chrome's Developer Tools to look at the returned headers and such and didn't gleam all these info.

👍

[SimonBaars]

So I figured out the root cause behind all of this: authentication is broken. All web calls done by this script fail, even though through CURL they work just fine. It might be that they introduced Cloudflare, or there's a different issue with auth.

[app4g]

CloudFlare has been there for some time already I believe. Either that or it's on for some user and not for some other users.

[SimonBaars]

Yeah, I'm not yet sure if Cloudflare is part of the problem. With CURL, my requests succeed. I'm now comparing the headers to see if there's one they may have added.

[SimonBaars]

Just to update on the progress: the old login page seems to set a different session cookie (called SESSION, but we need a different one called SESSIONID). This makes the follow-up requests fail.

Old login page: https://sso.garmin.com/sso/signin

New login page: https://sso.garmin.com/portal/sso/en-US/sign-in

I'm now trying to migrate it to the new login page which does set the correct cookies.

[app4g]

ah.. yes.. I also saw that, I was previously looking for SESSIONID as well, but like you said, now only SESSION is seen. I wished I had kept a screenshot of the headers when it was working to be able to compare the difference.

with respect to the login page, when I click the old login page link, and try to login, it will fail

the new login page, It's goes to a "Page Not Found" situation.

when I type connect.garmin.com into the safari, it redirects me to

https://connect.garmin.com/?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F

this works tho

https://sso.garmin.com/portal/sso/en-US/sign-in?clientId=GarminConnect&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern

[app4g]

I just tried to manually copy the SESSIONID cookie from the webBrowser after logging in and it worked. Essentially, you're right, we need to figure out how to get the SESSIONID cookie again.

I noticed that previously the SESSIONID was made available once we finish authentication (before sending the ticket back to Garmin) and thus we can store that. Now it's not there anymore but it's it one of the ticket URLs (I saw there are 2)

using HTTP Toolkit and browser session..

POST: https://sso.garmin.com/portal/api/login?clientId=GarminConnect&locale=en-GB&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern with these data { "username": "username@gmail.com", "password": "password", "rememberMe": false, "captchaToken": "axssfafa" }

Then GET: https://connect.garmin.com/modern?ticket=ST-2160819-kqmT7rZ6alD1pRbzV5QE-sso

Then GET: https://connect.garmin.com/modern/ This is where I start seeing the SESSIOIND cookie

Then GET: https://sso.garmin.com/sso/login?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&webhost=https%3A%2F%2Fconnect.garmin.com&gateway=true&generateExtraServiceTicket=true&generateTwoExtraServiceTickets=true&clientId=GarminConnect

then another Ticket GET request: https://connect.garmin.com/modern/?ticket=ST-0415002-je4Qp1iQO7ESbIBuWRQj-cas This has the same SESSIOIND cookie

Then GET: https://connect.garmin.com/modern/

[SimonBaars]

Yeah, SESSIONID is all we need. If we add this line after auth:

COOKIE_JAR.set_cookie(http.cookiejar.Cookie(version=0, name='SESSIONID', value='[your_session_id]', port=None, port_specified=False, domain='connect.garmin.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))

... the whole script runs through correctly and activities are exported.

As for obtaining the session id, the new auth flow sends a login POST:

curl 'https://sso.garmin.com/portal/api/login?clientId=GarminConnect&locale=en-US&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F' \
  -H 'content-type: application/json' \
  -H 'cookie: [your_cookies]' \
  --data-raw '{"username":"[uname]","password":"[pass]","rememberMe":false,"captchaToken":""}' \
  --compressed

If we copy the curl from Chrome, all is fine. If we script this, we get a 400 due to the lack of a captcha (even though it's empty in the curl version):

[ERROR] Server couldn't fulfill the request, url https://sso.garmin.com/portal/api/login?clientId=GarminConnect&locale=en-US&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern, code 400, error: HTTP Error 400: Bad Request
Content returned:
%s {"error":"com.garmin.sso.portal.service.ww.exception.InvalidReCaptchaException","errorText":"Recaptcha token is null/empty","errorItems":null,"conversationId":null}

Indeed, once we obtain the ticket from this request, we can use the ticket to obtain a session by calling https://connect.garmin.com/modern/?ticket=[your_ticket]. That request sets the session cookie, as seen in the devtools (or when doing curl verbose):

I've tried using Selenium to get the captcha to work, but somehow Garmin detects the use of browser automation. It seems that we're up against a quite robust login system. At least it doesn't have proper replay protection on abovementioned requests.

[SimonBaars]

As for a short-term solution, we could prompt the user for their SESSIONID token instead of uname/pass. Since that token expires, it's not a great solution for automation, what many users seem to use garmin-connect-export for.

I'll continue digging into ways to obtain a valid SESSIONID.

[app4g]

Funny thing tho, when I use (vanilla) chrome and try a login, it doesn't present me w/ a Captcha and I don't see / have any issues. But when I use HTTPToolKit and it spawns a Chrome session, then the captcha will turn up and I need to do the Captcha.

[SimonBaars]

I think Cloudflare decides how trustworthy your browser is, and part of its cookies is whether captcha validation can be skipped. In case of the chromedriver, it just completely rejects login (the login page loads, but the POST request gives 403).

[SimonBaars]

I created a draft version on my fork of a version that works by prompting for the session id: https://github.com/SimonBaars/garmin-connect-export/commit/6ed21656a008f395e65d24385c71ce70004c3dcb

It works, but unfortunately this approach appears to be flaky. As in, 1 out of 3 times it will fail due to being unauthenticated, but the behavior is completely inconsistent.

(anyone who would like to try it out: feel free, it should be able to export your activities, but you might need to re-run it if it fails)

[sgowtham]

Good morning.

May this offer some insight towards a potential solution?

https://github.com/petergardfjall/garminexport

Best regards, Gowtham (G)

-- Gowtham, PhD Director of Research Computing, IT Research Associate Professor, College of Computing Michigan Technological University

(906) 487-4096 https://it.mtu.edu https://hpc.mtu.edu

On Wed, Sep 27, 2023 at 6:18 AM Simon @.***> wrote:

By the way, the old login flow https://sso.garmin.com/sso/signin seems completely broken, even when not using automation.

— Reply to this email directly, view it on GitHub https://github.com/pe-st/garmin-connect-export/issues/95#issuecomment-1737116238, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXCPKCBOZDMWN7456TSO5TX4P4OJANCNFSM6AAAAAA5HL5DYE . You are receiving this because you commented.Message ID: @.***>

[SimonBaars]

@sgowtham Doesn't that have the same issue? https://github.com/petergardfjall/garminexport/issues/103

What might be interesting though is the Garth integration: https://github.com/petergardfjall/garminexport/issues/102

[philosowaffle]

Following along as I am also investigating how to get this resolved for my own project (p2g). While not a solution for me since my project is C#, Garth may be an option for y'all here. The library appears to use the Connect API to support authentication via the more standard OAuth. That being said, skimming through the initial login logic, I think this may hit the same error we are.

[sgowtham]

Yes - just tried out and ended up with the same issue. I thought the optional cloudfare during pip install might have had a built-in workaround but it didn't.

[dirac 07:02:05 ~]$ garmin-get-activity --destination GARMIN @. 12109737821 gpx Enter password: 2023-09-27 07:02:42,155 [INFO] authenticating user ... 2023-09-27 07:02:42,155 [INFO] fetching CSRF token ... 2023-09-27 07:02:43,266 [INFO] claiming auth ticket ... 2023-09-27 07:02:44,185 [INFO] fetching activity 12109737821 ... 2023-09-27 07:02:44,302 [ERROR] failed to fetch json summary for activity 12109737821: 403 {"message":"HTTP 403 Forbidden","error":"ForbiddenException"} 2023-09-27 07:02:44,302 [ERROR] failed with exception: failed to fetch json summary for activity 12109737821: 403 {"message":"HTTP 403 Forbidden","error":"ForbiddenException"} Traceback (most recent call last): File "/usr/local/bin/garmin-get-activity", line 8, in sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.11/site-packages/garminexport/cli/get_activity.py", line 68, in main summary = client.get_activity_summary(args.activity) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/garminexport/garminclient.py", line 73, in check_session return client_function(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/garminexport/garminclient.py", line 290, in get_activity_summary raise Exception(u"failed to fetch json summary for activity {}: {}\n{}".format( Exception: failed to fetch json summary for activity 12109737821: 403 {"message":"HTTP 403 Forbidden","error":"ForbiddenException"} [dirac 07:02:44 ~]$

Best regards, Gowtham (G)

-- Gowtham, PhD Director of Research Computing, IT Research Associate Professor, College of Computing Michigan Technological University

(906) 487-4096 https://it.mtu.edu https://hpc.mtu.edu

On Wed, Sep 27, 2023 at 7:01 AM Simon @.***> wrote:

@sgowtham https://github.com/sgowtham Doesn't that have the same issue? petergardfjall/garminexport#103 https://github.com/petergardfjall/garminexport/issues/103

— Reply to this email directly, view it on GitHub https://github.com/pe-st/garmin-connect-export/issues/95#issuecomment-1737175083, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXCPKCIZ57HXM42EA3HQN3X4QBQNANCNFSM6AAAAAA5HL5DYE . You are receiving this because you were mentioned.Message ID: @.***>

[SimonBaars]

I did some experimentation with Garth, and it seems like this is an approach that could work! It's able to correctly get oauth tokens and we can use those to get activities.

I'm working on a target implementation now.

[SimonBaars]

We have a candidate solution by migrating to Garth! PR is open ^^

https://github.com/pe-st/garmin-connect-export/pull/96

[wauwau0977]

Did get the same issue.

Yet, what I want to say: It's amazing to see how the community driven development works. A big thank you for working on this... 👍

[matin]

Garth maintainer here.

Just read through this thread. Amazing collaboration!

Garth took a lot of effort. Happy to hear it resolved the auth issues!

A recommendation for future dev: saving and reusing the OAuth tokens (instructions in the README). The OAuth1 token Garth obtains during the login is valid for a year. This is particularly important for people like me who use MFA.

Let me know how I can be helpful for using Garth or migrating it to other languages.

[dylix]

One thing I noticed from @SimonBaars https://github.com/SimonBaars/garmin-connect-export/tree/use-garth-oauth

(which I'm using now for my script :P)

URL_GC_ORIGINAL_ACTIVITY = 'http://connect.garmin.com/download-service/files/activity/' was unchanged, it still had the 'proxy/' in the URL.. removing it, and everything is working again!

What an amazing community thx so much for all the fixes!

[philosowaffle]

@matin Hi, thank you for the awesome work on Garth! I'm looking at porting this over to C# right now and came across this line. If I understand correctly, if Garmin revokes this key/secret pair then we'll all be broken again?

[matin]

@matin Hi, thank you for the awesome work on Garth! I'm looking at porting this over to C# right now and came across this line. If I understand correctly, if Garmin revokes this key/secret pair then we'll all be broken again?

The consumer key and secret are in S3 (instead of the code) to make sure they can be updated at any given time to ensure they're active.

[philosowaffle]

Makes sense. given Garmin's history of gatekeeping their API's to only "approved business developers", I'm concerned once they see how these creds are being used, they will revoke and not issue new ones.

Not saying we currently have a better option. Just documenting the risk.

[matin]

To be clear, these are not creds issued to me for business purposes. They're from the Garmin Connect app. They would only be updated if they're also updated in the app. In which case, I'll update with the new ones.

[SimonBaars]

@dylix Thanks for checking it out, I fixed the issue you mentioned!

[matin]

@SimonBaars I've said it before, but it bears repeating: great work on migrating other projects to Garth so quickly!

Given the broader use of garth.sso, I just released 0.4.30 (and now 0.4.31) to make the authentication more robust. The the OAuth1Session now uses the same retry and backoff logic as the session used to make requests.

Custom timeout and retry logic can be set with garth.configure().

Here are the defaults:

class Client:
    ...
    timeout: int = 10
    retries: int = 3
    status_forcelist: Tuple[int, ...] = (408, 429, 500, 502, 503, 504)
    backoff_factor: float = 0.5
    ....

[telemaxx]

@matin I have also the problem with our garminexport script.

so yesterday, I tried to install garth on debian Bookworm, but I have had no luck so far.

error was something about python environment. I will investigate further and report.

is garth working with python3?

[telemaxx]

i did this:

sudo apt install python3-pip

pip3 install garth --user

and i got a long error message:

$ pip3 install garth --user error: externally-managed-environment

× This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.

[matin]

@matin I have also the problem with our garminexport script.

so yesterday, I tried to install garth on debian Bookworm, but I have had no luck so far.

error was something about python environment. I will investigate further and report.

is garth working with python3?

Garth is available on Python 3.8 and higher. Try python -m pip install garth instead of pip3 install garth.

If you still run into issues, create a GH issue for Garth, and I can help you there.

[Haryt]

Hi, have the same Problem since 2 or 3 days. I installed garth (on windows 10, python 3.11, garmin export is the version from moderation) without success.

This is the errormessage:

ile "Z:\Sport\Script\gcexport.py", line 365, in ACTIVITY_LIST = http_req(URL_GC_LIST + urllib.parse.urlencode(SEARCH_PARAMS)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "Z:\Sport\Script\gcexport.py", line 165, in http_req response = OPENER.open(request, data=post) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\urllib\request.py", line 525, in open response = meth(req, response) ^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\urllib\request.py", line 634, in http_response response = self.parent.error( ^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\urllib\request.py", line 563, in error return self._call_chain(args) ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\urllib\request.py", line 496, in _call_chain result = func(args) ^^^^^^^^^^^ File "C:\Program Files\Python311\Lib\urllib\request.py", line 643, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

[telemaxx]

@matin I have also the problem with our garminexport script. so yesterday, I tried to install garth on debian Bookworm, but I have had no luck so far. error was something about python environment. I will investigate further and report. is garth working with python3?

Garth is available on Python 3.8 and higher. Try python -m pip install garth instead of pip3 install garth.

If you still run into issues, create a GH issue for Garth, and I can help you there.

Thank you for you're answer @matin.

i found a solution/workarround on Stackoverflow

as mentioned there i renamed one file(i dont want to delete it) sudo mv /usr/lib/python3.11/EXTERNALLY-MANAGED /usr/lib/python3.11/EXTERNALLY-MANAGED.OFF

than i was able to install garth: python3 -m pip install garth

next i have to implement garth in my garmin-connect-export fork that fork i a bit old, but with some extras which i need.

i created an issue at youre garth project, maybe you can give an installation hint in the docu.

[telemaxx]

Hi, have the same Problem since 2 or 3 days. I installed garth (on windows 10, python 3.11, garmin export is the version from moderation) without success.

did you mean the branch moderation?

Actually the PR with garth is not merged. so we have to wait a bit...

[Haryt]

Yes, i men the moderation branch, It seem that this branch is dead because since 1 year there is no change and also no answer to my question.... So I have to change to pe-st in the future.

[moderation]

Yes, i men the moderation branch, It seem that this branch is dead because since 1 year there is no change and also no answer to my question.... So I have to change to pe-st in the future.

@Haryt my moderation branch isn't dead. I've had a recent cycling injury and am still getting back up to speed. I will incorporate the changes from this branch into mine shortly.

[Haryt]

@moderation Good to hear that the branch is living. Whish you all the best for recovery!

[app4g]

FYI, the folks here https://github.com/petergardfjall/garminexport/issues/103#issuecomment-1742121659 has managed to get it sorted out w/o using garth.

[pe-st]

Should be fixed with PR #96 (merged into v4.2.0)

[moderation]

@moderation Good to hear that the branch is living. Whish you all the best for recovery!

@Haryt I've finally gotten around to updating my fork. I borrowed heavily from @pe-st work with Garth. I just tested on my last 3 rides and it seems to be working.

https://github.com/moderation/garmin-connect-export/