search

tcgoetz / GarminDB

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
GNU General Public License v2.0
1.1k stars 138 forks source link

Use connectapi.garmin.com API to make Garth authentication work #196

Closed msiemens closed 11 months ago

msiemens commented 11 months ago

This makes file/data downloads work with Garth's authentication. The main insight that allowed this was that the connect.garmin.com/modern/proxy URLs were just a proxy to connectapi.garmin.com. The lead was this:

https://github.com/yihong0618/running_page/blob/22966d8546d424cadc92f8819804c56a736fcee0/run_page/garmin_sync.py#L30

This also uses Garth but uses the connectapi.garmin.com domain to fetch data. For instance, it uses connectapi.garmin.com/activitylist-service/activities/search/activities instead of connect.garmin.com/proxy/activitylist-service/activities/search/activities. Testing this URL with the authentication token retrieved by Garth returned valid data, indicating that connect.garmin.com/proxy has been removed in favor of connectapi.garmin.com.

See #192

msiemens commented 11 months ago

Note: I got a few errors when running tests but cannot tell if they are caused by these changes or not:

$ make test
$PROJECT_BASE is [/Users/markus/tmp/gdb/GarminDB]
$PLATFORM is [Darwin]
$SHELL is [/bin/sh]
$PIP_PATH is [/Users/markus/tmp/gdb/GarminDB/.venv/bin/pip3]
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C Fit test
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C test
python3 test_fit_fields.py
test_local_timestamp_field_valid_conversion (__main__.TestFitFields.test_local_timestamp_field_valid_conversion) ... ok
test_time_ms_field_valid_conversion (__main__.TestFitFields.test_time_ms_field_valid_conversion) ... ok
test_utc_timestamp_field_valid_conversion (__main__.TestFitFields.test_utc_timestamp_field_valid_conversion) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK
python3 test_fit_field_enum.py
test_enum_field_unknown_conversion (__main__.TestFitFieldEnum.test_enum_field_unknown_conversion) ... ok
test_enum_field_valid_conversion (__main__.TestFitFieldEnum.test_enum_field_valid_conversion) ... ok
test_field_enum_fuzzy_metric (__main__.TestFitFieldEnum.test_field_enum_fuzzy_metric) ... ok
test_field_enum_fuzzy_statute (__main__.TestFitFieldEnum.test_field_enum_fuzzy_statute) ... ok
test_field_enum_unknown_conversion (__main__.TestFitFieldEnum.test_field_enum_unknown_conversion) ... ok
test_field_enum_valid_conversion (__main__.TestFitFieldEnum.test_field_enum_valid_conversion) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK
python3 test_fit_dependant_field.py
test_product_field_reconvert (__main__.TestFitDependantField.test_product_field_reconvert) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
python3 test_measurements.py
test_distance (__main__.TestMeasurement.test_distance) ... ok
test_distance_from_func (__main__.TestMeasurement.test_distance_from_func) ... ok
test_distance_from_func_raises (__main__.TestMeasurement.test_distance_from_func_raises) ... ok
test_speed_from_func (__main__.TestMeasurement.test_speed_from_func) ... ok
test_speed_from_func_raises (__main__.TestMeasurement.test_speed_from_func_raises) ... ok
test_temperature (__main__.TestMeasurement.test_temperature) ... ok
test_weight (__main__.TestMeasurement.test_weight) ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.000s

OK
python3 test_conversions.py
test_perhour_speed_to_pace (__main__.TestConversions.test_perhour_speed_to_pace) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C Tcx test
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C test
python3 test_loop.py
test_loop (__main__.TestLoop.test_loop) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
python3 test_read.py
test_parse_tcx (__main__.TestRead.test_parse_tcx) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C utilities test
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C test
make[2]: Nothing to be done for `all'.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C test all
python3 test_garmin_db.py
db params <DbParams() {'db_type': 'sqlite', 'db_path': '/var/folders/mx/712zg2513_b0p2hxxw03bq480000gn/T/tmpu4514lyk/DBs'}
test_db_exists (__main__.TestGarminDb.test_db_exists) ... ok
test_db_tables_exists (__main__.TestGarminDb.test_db_tables_exists) ... FAIL
test_garmindb_tables_bounds (__main__.TestGarminDb.test_garmindb_tables_bounds) ... ERROR
test_measurement_system (__main__.TestGarminDb.test_measurement_system) ... ok
test_not_none_cols (__main__.TestGarminDb.test_not_none_cols) ... ok
test_sleep_import (__main__.TestGarminDb.test_sleep_import) ... Processing [<FileType.sleep: 49>] FIT data from ./garmindb/test_files/fit/sleep
ERROR

======================================================================
ERROR: test_garmindb_tables_bounds (__main__.TestGarminDb.test_garmindb_tables_bounds)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/markus/tmp/gdb/GarminDB/test/test_garmin_db.py", line 77, in test_garmindb_tables_bounds
    self.check_col_stats(
  File "/Users/markus/tmp/gdb/GarminDB/test/test_garmin_db.py", line 62, in check_col_stats
    self.check_col_stat(col_name + ' max', maximum, max_bounds)
  File "/Users/markus/tmp/gdb/GarminDB/test/test_garmin_db.py", line 51, in check_col_stat
    self.assertGreaterEqual(value, min_value, '%s value %s less than min %s' % (value_name, value, min_value))
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/unittest/case.py", line 1275, in assertGreaterEqual
    if not a >= b:
           ^^^^^^
TypeError: '>=' not supported between instances of 'NoneType' and 'int'

======================================================================
ERROR: test_sleep_import (__main__.TestGarminDb.test_sleep_import)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/markus/tmp/gdb/GarminDB/test/test_garmin_db.py", line 125, in test_sleep_import
    gfd = GarminSleepFitData('./garmindb/test_files/fit/sleep', latest=False, measurement_system=self.measurement_system, debug=2)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/markus/tmp/gdb/GarminDB/.venv/lib/python3.11/site-packages/garmindb/import_monitoring.py", line 93, in __init__
    super().__init__(input_dir, debug, latest, True, [fitfile.FileType.sleep], measurement_system)
  File "/Users/markus/tmp/gdb/GarminDB/.venv/lib/python3.11/site-packages/garmindb/fit_data.py", line 41, in __init__
    self.file_names = FileProcessor.dir_to_files(input_dir, fitfile.file.name_regex, latest, recursive)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/markus/tmp/gdb/GarminDB/.venv/lib/python3.11/site-packages/idbutils/file_processor.py", line 46, in dir_to_files
    for file in os.listdir(input_dir):
                ^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './garmindb/test_files/fit/sleep'

======================================================================
FAIL: test_db_tables_exists (__main__.TestGarminDb.test_db_tables_exists)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/markus/tmp/gdb/GarminDB/test/test_db_base.py", line 46, in test_db_tables_exists
    self.check_db_tables_exists(self.db, self.table_dict)
  File "/Users/markus/tmp/gdb/GarminDB/test/test_db_base.py", line 39, in check_db_tables_exists
    self.assertGreaterEqual(table.row_count(db), min_rows, 'table %s has no data' % table_name)
AssertionError: 0 not greater than or equal to 1 : table attributes_table has no data

----------------------------------------------------------------------
Ran 6 tests in 0.022s

FAILED (failures=1, errors=2)
make[1]: *** [garmin_db] Error 1
make: *** [test] Error 2
msiemens commented 11 months ago

Ahh, just for sake of completeness: this needs tcgoetz/utilities#4 to be merged to work properly

tcgoetz commented 11 months ago

I looking in to it, but "table attributes_table has no data" probably means that the new download code isn't downloading everything that the old did.