search

La0 / garmin-uploader

Garmin Connect Python Uploader
GNU General Public License v2.0
124 stars 30 forks source link

Authentication Failures again #28

Open philosowaffle opened 3 years ago

philosowaffle commented 3 years ago

Howdy,

Seems Garmin has yet again tweaked some things and I have several users (including myself) running into auth failures. https://github.com/philosowaffle/peloton-to-garmin/issues/109

assert self.user.authenticate(), "Failed to authenticate garmin user."
AssertionError: Failed to authenticate garmin user.

Thank you in advance for any help!

aurelg commented 3 years ago

⚠ possibly dumb idea: could browser automation (e.g. w/Selenium) help?

dkmcgowan commented 3 years ago

Making things more complicated but if we did get a license, we could host an API in between in AWS or something, so a sync to Garmin service that would reuse the creds because they would stay in the backend. That all costs monthly money though, probably not much, but something to manage

philosowaffle commented 3 years ago

Garmin has updated their site to indicate API access is free now but the request form is highly geared towards businesses. Doesn't mean they wouldn't grant us access, just seems highly unlikely. Especially, when you look at existing paid services like SyncMyWorkout and Tapiriik, neither of which are using the official API either. Garmin is likely still looking for a way to partner with businesses like Peloton, not further enable the community to move its data around freely. 😢

kenlpeters commented 3 years ago

I wonder if they allowing activity by Ip whitelist. Maybe that is why AWS server is working because they have Amazons servers open their API. Just a theory.

superm1 commented 3 years ago

If that's the case, maybe also authenticating and interacting in a browser as a user from your IP will whitelist you?

kenlpeters commented 3 years ago

Maybe i have not signed up so I am not sure what questions they ask. If one is what is the IP address your application will making calls from, then it definitely can be a white listing component.

philosowaffle commented 3 years ago

Two things I tested just now that seems to be working:

  1. I updated Python to 3.9.5 (this may be irrelevant but want to mention just in case)
  2. I checked out and ran the cloudscraper branch.

For 1 - I chose to do this after reading through this linked issue

For 2 - I want to note that I may have run the wrong branch when I previously attempted to test the cloudscraper fix. But for the life of me I cannot remember. Sorry for the confusion, but it does seem to be working now.

Oddly enough, I tried 5 times, it worked on 4 attempts, failed on 1.

G:\Code\Git\garmin-uploader>git status
On branch cloudscraper
Your branch is up-to-date with 'origin/cloudscraper'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        gupload.ini

nothing added to commit but untracked files present (use "git add" to track)

G:\Code\Git\garmin-uploader>pip install .    
Processing g:\code\git\garmin-uploader
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Collecting cloudscraper==1.2.58
  Using cached cloudscraper-1.2.58-py2.py3-none-any.whl (96 kB)
Requirement already satisfied: six>=1.10.0 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from garmin-uploader==1.0.10) (1.16.0)
Requirement already satisfied: requests>=2.9.2 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages 
(from cloudscraper==1.2.58->garmin-uploader==1.0.10) (2.25.1)
Collecting pyparsing>=2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting requests-toolbelt>=0.9.1
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (2.10)
Using legacy 'setup.py install' for garmin-uploader, since package 'wheel' is not installed.
Installing collected packages: requests-toolbelt, pyparsing, cloudscraper, garmin-uploader
  Attempting uninstall: garmin-uploader
    Found existing installation: garmin-uploader 1.0.9
    Uninstalling garmin-uploader-1.0.9:
      Successfully uninstalled garmin-uploader-1.0.9
    Running setup.py install for garmin-uploader ... done
Successfully installed cloudscraper-1.2.58 garmin-uploader-1.0.10 pyparsing-2.4.7 requests-toolbelt-0.9.1

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 14:56:49,964 [INFO] Try to login on GarminConnect...
2021-05-26 14:56:51,867 [INFO] Logged in as bbdevelopertest01@gmail.com
2021-05-26 14:56:53,224 [INFO] Uploaded activity 6847673340 : sample_file.tcx
2021-05-26 14:56:53,224 [INFO] All done.

G:\Code\Git\garmin-uploader>python --version
Python 3.9.5

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 14:59:30,359 [INFO] Try to login on GarminConnect...
2021-05-26 14:59:32,173 [CRITICAL] Login Failure: Login check failed.
Error: Invalid credentials

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 14:59:49,262 [INFO] Try to login on GarminConnect...
2021-05-26 14:59:51,266 [INFO] Logged in as bbdevelopertest01@gmail.com
2021-05-26 14:59:51,776 [INFO] Activity already uploaded 6847673340 : sample_file.tcx
2021-05-26 14:59:51,776 [INFO] All done.

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 15:00:11,777 [INFO] Try to login on GarminConnect...
2021-05-26 15:00:14,012 [INFO] Logged in as bbdevelopertest01@gmail.com
2021-05-26 15:00:15,255 [INFO] Activity already uploaded 6847673340 : sample_file.tcx
2021-05-26 15:00:15,255 [INFO] All done.

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 15:00:34,871 [INFO] Try to login on GarminConnect...
2021-05-26 15:00:36,780 [INFO] Logged in as bbdevelopertest01@gmail.com
2021-05-26 15:00:37,272 [INFO] Activity already uploaded 6847673340 : sample_file.tcx
2021-05-26 15:00:37,272 [INFO] All done.

G:\Code\Git\garmin-uploader>

Other things I tested during this session (on the fix-cloudfare-errors branch):

  1. Using a VPN tested in Houston, TX, US West Coast, France -> auth failures in all cases
  2. I tried changing the UserAgent used in gupload -> no impact
  3. maybe authenticating and interacting in a browser as a user from your IP will whitelist you - my dev box is my personal pc, so I am regularly logged into Garmin Connect in the browser. I also test and work on a headless RPi4, up until just now, script authentication has failed on both.

I will continue to tentatively test the cloudscraper branch.... am a little paranoid of drawing attention myself and getting shut out again lol

La0 commented 3 years ago

What a mess... I'm using the last release myself and have no issues to upload activities.

I think they have some anti-cheat heuristics on a few factors:

I can't find how to get the geolocation argument from garmin (basically to tell if you are en-us or fr-fr, etc...)

Another attempt would be to try something like selenium, but that would be really heavy and hard to maintain too :disappointed:

I don't have a silver bullet for now.

kenlpeters commented 3 years ago

Just an update from my end. I am using the cloud fix CI with V1 peloton-to-Garmin and it continues to work on my MS Azure Linux VM. I am using python 3.7 and my URLs are fr-FR. I apologized I have been saying my VM is an AWS one when it is an Azure in Central US.

kenlpeters commented 3 years ago

Let me know if there is anything I can test for you.

dkmcgowan commented 3 years ago

If you update v2 with cloudscrapper branch I'll pull it down and test

dkmcgowan commented 3 years ago

I'm the other comments can we just add a param for country code we manually put in and then add a delay for the login request and give that a shot

philosowaffle commented 3 years ago

@dkmcgowan I have updated the V2 docker image to use the cloudscraper fix. You can pull latest and test it out. Seems to still be working for me.

dkmcgowan commented 3 years ago

This one is working for me, atleast the first run did to upload a few

philosowaffle commented 3 years ago

I've integrated the CloudScraper fix into my latest build and am taking a poll (see last comment) to see who its working for.

philosowaffle commented 3 years ago

So far the cloudscraper fix seems to be working for all of my users who have reported back (~10 so far). No one has reported that it is not working for them.

zensonic commented 3 years ago

I have just downloaded, compiled and installed python3.9 on raspbian.. that made things work again using gupload

2021-05-29 06:03:34,133 [INFO] Force uploading 2021-05-29 06:03:34,136 [INFO] - device 3895796269 2021-05-29 06:03:34,151 [INFO] Going to upload 368 activities to garmin connect 2021-05-29 06:07:03,077 [INFO] Uploaded files to garmin connect

zed4 commented 3 years ago

cloudscraper fix seems to work for me as well. (i'm using this from within peloton-to-garmin)

mahboc commented 3 years ago

FWIW- Standard Installation on Mac Big Sur (11.13.1) Python 3.8 fails authentication.

mahboc commented 3 years ago

Following directions (cloudscraper) above seems to fix the issue for me.

jmodena commented 3 years ago

garmin-uploader 1.0.10 in Virtualbox machine with Ubuntu 20.04 (python 3.8.5) works perfectly.

philosowaffle commented 3 years ago

@La0 - Question when you have a chance. My project has switched to using the cloudscraper fix with moderate success for most users. One issue we are seeing is when we attempt upload multiple activities at once. If we attempt to only upload 1 activity, then the upload succeeds. However, attempting to upload more than 1 at once results in an auth error with Garmin.

Below is a sample command that is executed. Email and Password are scrubbed intentionally:

C:\Users\mikel\OneDrive\Desktop\p2g_v2\python\upload\upload.exe -ge **email** -gp **password** -f "C:\Users\mikel\OneDrive\Desktop\p2g_v2\working\upload\0667494098494e6e891f97d3afa1d9a6_20_min_Low_Impact_Ride_with_Jenn_Sherman.fit" "C:\Users\mikel\OneDrive\Desktop\p2g_v2\working\upload\c92726553fb7431db53375943b0edbd5_20_min_Rock_Ride_with_Denis_Morton.fit" "C:\Users\mikel\OneDrive\Desktop\p2g_v2\working\upload\ccddfdc691e74e44b7256660e6b37796_30_min_2000s_Ride_with_Hannah_Frankson.fit"

Here is a comment with more details on repro steps.

I'm wondering if something changed in the cloudscraper fix where it attempts to authenticate more than once when uploading multiple files? Or if there is something I should be doing differently in these cases?

PeregrineFalcon commented 3 years ago

Want to report that using 1.0.10 is working on a Heroku install as a library dependency within https://github.com/flackdl/ifit-garmin-sync

plandes commented 3 years ago

Hello. I am trying to incorporate this change as well, and resorted to adding api.py to my own project outright. However, I'm getting a 403 using it verbatim (with a small change of using module logging for the logger). Does anyone have any idea why I'd be getting a 403 (forbidden)?

Thanks in advance.

hugoideler commented 2 years ago

Hi! My experience is that the cloudscraper branch is working well, while the master branch has consistently failed (always resulting in the 'authentication failed' misnomer errors). In this thread I'm also reading a lot of positive feedback on this branch.

Would it be possible to merge the cloudscraper branch into master and bump the library version?

kenlpeters commented 2 years ago

Checking in here. Although the PelotonToGarminConsole is working wonderfully, I have started a new process of syncing iFit (for Norditrack) to Garmin. I am using the uploader again and it continues to get the too many requests error.

app4g commented 1 year ago

I am getting auth errors with this and many other projects. It fails at getting the CSRF token/ticket.

I tried using https://github.com/felipeam86/garpy/tree/4ced33f4dcf018278e948b6eaea1ab6414d2ef5a and it worked.

After some looking at it, it seems that one of the difference may be where the response token is being looked for?

https://connect.garmin.com/modern?ticket=ST-0123456-aBCDefgh1iJkLmN5opQ9R-cas

coming from this method/function here https://github.com/felipeam86/garpy/blob/4ced33f4dcf018278e948b6eaea1ab6414d2ef5a/garpy/client.py#L35

Stoobiedoo commented 1 year ago

After a year of working with the crowdscraper code, it is now resulting in 403 errors. Is anyone else running into connection troubles starting this week?

app4g commented 1 year ago

yeah.. its down for most projects...

https://github.com/lswiderski/mi-scale-exporter/issues/20#issuecomment-1736213173

oldnapalm commented 1 year ago

garth is working

https://github.com/oldnapalm/garmin-uploader/commit/7234be792f152d4bdb5e14d5ab747298d8e3763d

pantomnesia commented 1 year ago

After a year of working with the crowdscraper code, it is now resulting in 403 errors. Is anyone else running into connection troubles starting this week?

Same here. Simple test from cli

gupload -u myusername -p mypassword myfile.tcx

results in.....

2023-10-05 11:56:57,277 [INFO] Try to login on GarminConnect...
2023-10-05 11:57:00,300 [CRITICAL] Login Failure: Expecting value: line 1 column 1 (char 0)

`

oldnapalm commented 1 year ago

@pantomnesia see https://github.com/oldnapalm/garmin-uploader/commit/7234be792f152d4bdb5e14d5ab747298d8e3763d (notice that it will only fix authentication but you will still get upload error and updating activity name and type won't work because detailedImportResult no longer contains internalId https://github.com/oldnapalm/garmin-uploader/commit/8344c0e77a91497c8a1a04ede4c87db68146f29a).

Or just migrate to https://github.com/matin/garth

Example:

import os
import sys
import garth
from getpass import getpass

if len(sys.argv) < 2:
    print("Use %s <file name>" % sys.argv[0])
    sys.exit(1)

if not os.path.isfile(sys.argv[1]):
    print("File %s not found" % sys.argv[1])
    sys.exit(1)

tokens_dir = '~/.garth'

try:
    garth.resume(tokens_dir)
    garth.client.username
except:
    email = input("Enter email address: ")
    password = getpass("Enter password: ")
    try:
        garth.login(email, password)
        garth.save(tokens_dir)
    except Exception as exc:
        print(exc)
        sys.exit(1)

try:
    with open(sys.argv[1], "rb") as f:
        resp = garth.client.upload(f)
        print(resp)
except Exception as exc:
    print(exc)
    sys.exit(1)

sys.exit(0)