search

tcgoetz / GarminDB

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
GNU General Public License v2.0
1.18k stars 142 forks source link

recent change on download_garmin.py? #19

Closed benbowen closed 6 years ago

benbowen commented 6 years ago

The past two days, my history failed to download. Did Garmin change their API? Are other people able to download their monitoring files? I checked through a browser and I can login fine with the credentials I'm using.

The command (called from make):

python download_garmin.py -l --sqlite /Users/bpb/HealthData/DBs -u <name> -p <pass> -m "/Users/bpb/HealthData/FitFiles/2018_Monitoring"

The traceback:

Traceback (most recent call last):
  File "download_garmin.py", line 441, in <module>
    main(sys.argv[1:])
  File "download_garmin.py", line 389, in main
    download.login(username, password)
  File "download_garmin.py", line 132, in login
    response = self.get(self.garmin_connect_modern_url, params)
  File "download_garmin.py", line 72, in get
    response.raise_for_status()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/requests/models.py", line 939, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://connect.garmin.com/modern?ticket=ST-2475442-cyLN5naZfOHIzOXA2gMa-cas
make: *** [download_new_monitoring] Error 1
benbowen commented 6 years ago

Any ideas @tcgoetz ? I did a conda update --all for better or worse (no effect for this bug). My computer also did an auto-restart and installed latest Mac OS updates. These updates made me change the way my VPN software worked. Looking at the header for login def, I can imagine the problem is Garmin changed something there.

tcgoetz commented 6 years ago

Sorry. I've been away. I hit this last week and made some changes to login. It's working for me now. Get the latest and see if your still having issues.

benbowen commented 6 years ago

No worries! You are amazing for putting this thing together!

I'm worried that I should not have done "update -all"

It looks like my version of sqlalchemy is freaking out.

INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Debug: 0 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Importing /Users/bpb/HealthData/FitFiles/2018_Monitoring//21685089764.fit (2018-06-16 19:04:59) [FileType.monitoring_b] with message types: [<MessageType.file_id: 0>, <MessageType.device_info: 23>, <MessageType.software: 35>, <MessageType.monitoring_info: 103>, <MessageType.monitoring: 55>, <UnknownMessageType.unknown_211: 211>]

Traceback (most recent call last):
  File "import_garmin.py", line 301, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 287, in main
    gfd.process_files(db_params_dict)
  File "import_garmin.py", line 79, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 67, in write_file
    self.write_message_types(fit_file, fit_file.message_types())
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 59, in write_message_types
    self.write_message_type(fit_file, message_type)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 46, in write_message_type
    function(fit_file, message_type, messages)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 38, in write_generic
    function(fit_file, message)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 339, in write_monitoring_info_entry
    GarminDB.MonitoringInfo.find_or_create(self.garmin_mon_db, entry)
  File "/Users/bpb/repos/garmin_activity/GarminDB/HealthDB/DB.py", line 231, in find_or_create
    DB.commit(session)
  File "/Users/bpb/repos/garmin_activity/GarminDB/HealthDB/DB.py", line 58, in commit
    session.commit()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
    self._flush(objects)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
    flush_context.execute()
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 830, in _emit_insert_statements
    execute(statement, multiparams)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/bpb/anaconda2/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: monitoring_info.file_id [SQL: u'INSERT INTO monitoring_info (timestamp, file_id, activity_type, resting_metabolic_rate, cycles_to_distance, cycles_to_calories) VALUES (?, ?, ?, ?, ?, ?)'] [parameters: ('2018-06-16 19:05:00.000000', None, 'walking', 2784, 1.829, 0.09183998724998008)] (Background on this error at: http://sqlalche.me/e/gkpj)
make: *** [import_monitoring] Error 1
tcgoetz commented 6 years ago

file_id is None which shouldn't happen. If you run:

make TEST_GC_ID=21685089764 test_import_fit_activities

what do the logs just before the exception say? Can you paste them in here? You must be getting an exception in write_file_id_entry.

tcgoetz commented 6 years ago

Probably looking for the output of htis line:

                logger.debug("No entry handler %s for message type %s (%d) from %s: %s",
                    handler_name, repr(message_type), len(messages), fit_file.filename, str(messages[0]))
benbowen commented 6 years ago

Here it is. Whenever I go to my kick-boxing class, the Garmin watch often automatically starts either swimming or an indoor cycling activity; so likely the activity being parsed is one of those two activity types.

There is a lengthy log of battery, system settings, user-profile, position, heart-rate, cadence messages. The log ends with a message and traceback like you say above.

DEBUG:Fit.File:  Message [6]: DataMessage: <UnknownMessageType.unknown_113: 113>: unknown_4(893258310 (893258310)), unknown_5(0 (0)), unknown_2(100000 (100000)), unknown_3(1343662 (1343662)), unknown_0(0 (0)), unknown_1(1 (1)), timestamp(2018-04-21 08:41:00 (893259661)), 
DEBUG:Fit.File:File /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: 2018-04-21 08:01:13 -> 2018-04-21 08:41:00
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Importing /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit (2018-04-21 08:01:13) [FileType.activity] with message types: [<MessageType.file_id: 0>, <MessageType.file_creator: 49>, <MessageType.event: 21>, <MessageType.device_info: 23>, <MessageType.source: 22>, <MessageType.unknown_141: 141>, <MessageType.device_settings: 2>, <MessageType.user_profile: 3>, <UnknownMessageType.unknown_79: 79>, <MessageType.sport: 12>, <MessageType.unknown_13: 13>, <MessageType.zones_target: 7>, <MessageType.record: 20>, <MessageType.lap: 19>, <UnknownMessageType.unknown_216: 216>, <MessageType.unknown_140: 140>, <MessageType.session: 18>, <MessageType.activity: 34>, <UnknownMessageType.unknown_113: 113>]
DEBUG:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:No entry handler write_file_id_entry for message type <MessageType.file_id: 0> (1) from /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: DataMessage: <MessageType.file_id: 0>: product(GarminProduct.VivoActive_HR (2337)), product(GarminProduct.VivoActive_HR (2337)), time_created(2018-04-21 08:01:13 (893257274)), number(None (65535) [invalid]), serial_number(3935780983 (3935780983)), type(FileType.activity (4)), product_name(4294967295 (4294967295) [invalid]), manufacturer(Manufacturer.Garmin (1)), 
DEBUG:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Processed 1 <MessageType.file_id: 0> entries for /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit
DEBUG:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:No entry handler write_device_info_entry for message type <MessageType.device_info: 23> (12) from /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: DataMessage: <MessageType.device_info: 23>: battery_voltage(None v (65535) [invalid]), device_type(None (255) [invalid]), unknown_8(None (4294967295) [invalid]), unknown_9(None (255) [invalid]), unknown_13(None (65535) [invalid]), unknown_15(None (4294967295) [invalid]), unknown_16(None (4294967295) [invalid]), unknown_17([None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None] ([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])), product(GarminProduct.VivoActive_HR (2337)), cum_operating_time(None (4294967295) [invalid]), device_index(0 (0)), ant_transmission_type(None (0) [invalid]), serial_number(3935780983 (3935780983)), product(GarminProduct.VivoActive_HR (2337)), software_version(4.3 (430)), timestamp(2018-04-21 08:01:13 (893257274)), source_type(SourceType.local (5)), ant_device_number(None (0) [invalid]), unknown_24(None (0) [invalid]), unknown_23(None (255) [invalid]), manufacturer(Manufacturer.Garmin (1)), hardware_version(None (255) [invalid]), battery_status(None (255) [invalid]), ant_network(None (255) [invalid]), 
WARNING:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:device_info: {'battery_voltage': None, 'device_type': 4, 'unknown_8': None, 'unknown_9': None, 'unknown_13': None, 'unknown_15': None, 'unknown_16': None, 'unknown_17': [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], 'Product': <GarminProduct.VivoActive_HR: 2337>, 'cum_operating_time': None, 'device_index': 1, 'ant_transmission_type': None, 'serial_number': None, 'product': <GarminProduct.VivoActive_HR: 2337>, 'software_version': 4.3, 'timestamp': datetime.datetime(2018, 4, 21, 8, 1, 13), 'source_type': <SourceType.local: 5>, 'ant_device_number': None, 'unknown_24': None, 'unknown_23': None, 'manufacturer': <Manufacturer.Garmin: 1>, 'hardware_version': None, 'battery_status': None, 'ant_network': None} - 'int' object has no attribute 'value'
ERROR:import_garmin_activities.py:Failed to parse /Users/bpb/HealthData/FitFiles/Activities/2640913347.fit: Failed to translate device_info
Traceback (most recent call last):
  File "import_garmin_activities.py", line 399, in <module>
    main(sys.argv[1:])
  File "import_garmin_activities.py", line 395, in main
    gfd.process_files(db_params_dict)
  File "import_garmin_activities.py", line 45, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 67, in write_file
    self.write_message_types(fit_file, fit_file.message_types())
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 56, in write_message_types
    self.write_message_type(fit_file, message_type)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 46, in write_message_type
    function(fit_file, message_type, messages)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 38, in write_generic
    function(fit_file, message)
  File "/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.py", line 376, in write_device_info_entry
    raise ValueError('Failed to translate device_info')
ValueError: Failed to translate device_info
make: *** [test_import_fit_activities] Error 1
benbowen commented 6 years ago

I'm fairly confident that this error is because I was out of sync between my Fit rep and GarminDB repo. I did a git pull on Fit and was able to get through a ton of activity files without error.