search

tcgoetz / GarminDB

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
GNU General Public License v2.0
1.18k stars 142 forks source link

Exception on entry #13

Closed umaar closed 6 years ago

umaar commented 6 years ago

After running this command: make GC_USER='username' GC_PASSWORD='password'

I can see it starts populating the database, but I see a lot of these errors in the terminal during execution, any ideas? 

INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:Exception on entry: {'timestamp': datetime.datetime(2018, 2, 16, 14, 41, 59), 'unknown_38': 0, 'unknown_37': 0}: 'unknown_38'
INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:/Users/umarhansa/HealthData/FitFiles/2018_Monitoring//21375899400.fit [monitoring_b] message types: ['file_id', 'device_info', 'software', 'monitoring_info', 'monitoring', 'unknown_msg_211', 'event', 'unknown_msg_24']
INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:Exception on entry: {'timestamp': datetime.datetime(2018, 6, 7, 0, 52, 59), 'unknown_38': 0, 'unknown_37': 0}: 'unknown_38'
INFO:/Users/umarhansa/Downloads/GarminDB/FitFileProcessor.pyc:Exception on entry: {'timestamp': datetime.datetime(2018, 6, 7, 16, 52, 59), 'unknown_38': 0, 'unknown_37': 0}: 'unknown_38'

Really cool project by the way!

tcgoetz commented 6 years ago

That's expected. "Exception on entry" is saying that a monitoring entry has undocumented fields 37 and 38. The other log line is saying that the file as undocumented messages with Ids 24 and 211.

If I can get the field and message formats, I can add them and those messages with be decoded. Garmin uses a bunch of messages and fields that are not detailed int he SDK.

umaar commented 6 years ago

Thank you for explaining!

benbowen commented 6 years ago

Yesterday, I pulled and rebuilt my database. I also get the 24 and 211 "unknown_msg_X" INFO printed hundreds of times.

Is there something I can do to help pinpoint the missing attributes and values in the fit files? If yes, I will need some help getting started with that. The log tells me the fit files which contain the unexpected field names.

tcgoetz commented 6 years ago

@benbowen There are make targets test_import_activities and test_monitoring_file that allow you to specify a single FIT file with TEST_FIT_FILE or TEST_ACTIVITY_ID. They dump the corresponding FIT file with debugging turned up. You can file a bug with the log line for the unknown message. If there are multiple samples of that message with different values, please include a few.

benbowen commented 6 years ago

Sorry @tcgoetz, but I can't figure out the syntax to specify a fit file. I tried setting environment variables of "TEST_FILE_FILE" and "TEST_ACTIVITY_ID" to a filename, but when I execute "make test_import_activities" or "make test_monitoring_file" a different fit file is used. I'm doing this wrong....

Here is terminal output:

[bpb@Benjamins-MacBook-Pro-4 ~ 11:49 AM] ~/repos/garmin_activity/GarminDB (master) > export TEST_ACTIVITY_ID=24955272738
[bpb@Benjamins-MacBook-Pro-4 ~ 11:54 AM] ~/repos/garmin_activity/GarminDB (master) > make test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit" --sqlite /Users/bpb/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 1
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
DEBUG:HealthDB.DB:DB {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug 0 
INFO:GarminDB.MonitoringDB:MonitoringDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:GarminDB.ActivitiesDB:ActivitiesDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Debug: 1 English units: True
Traceback (most recent call last):
  File "import_garmin_activities.py", line 401, in <module>
    main(sys.argv[1:])
  File "import_garmin_activities.py", line 397, in main
    gfd.process_files(db_params_dict)
  File "import_garmin_activities.py", line 43, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/Fit/File.py", line 30, in __init__
    self.file = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit'
make: *** [test_import_activities] Error 1
[bpb@Benjamins-MacBook-Pro-4 ~ 11:54 AM] ~/repos/garmin_activity/GarminDB (master) > export TEST_FIT_FILE=24955272738
[bpb@Benjamins-MacBook-Pro-4 ~ 11:56 AM] ~/repos/garmin_activity/GarminDB (master) > make test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit" --sqlite /Users/bpb/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 1
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:import_garmin_activities.py:Debug: 1 English units: True
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FileProcessor.pyc:Reading file: /Users/bpb/HealthData/FitFiles/Activities/1589795363.fit
INFO:GarminDB.GarminDB:GarminDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
DEBUG:HealthDB.DB:DB {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug 0 
INFO:GarminDB.MonitoringDB:MonitoringDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:GarminDB.ActivitiesDB:ActivitiesDB: {'db_type': 'sqlite', 'db_path': '/Users/bpb/HealthData/DBs'} debug: 0 
INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:Debug: 1 English units: True
Traceback (most recent call last):
  File "import_garmin_activities.py", line 401, in <module>
    main(sys.argv[1:])
  File "import_garmin_activities.py", line 397, in main
    gfd.process_files(db_params_dict)
  File "import_garmin_activities.py", line 43, in process_files
    fp.write_file(Fit.File(file_name, self.english_units))
  File "/Users/bpb/repos/garmin_activity/GarminDB/Fit/File.py", line 30, in __init__
    self.file = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: '/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit'
make: *** [test_import_activities] Error 1
tcgoetz commented 6 years ago

@benbowen because they are Make variables not env variables.

Try: make TEST_ACTIVITY_ID=24955272738 test_import_activities

I will push a commit to clean up running a single file in debug to get this info soon.

benbowen commented 6 years ago

Still not getting it. Here is info log when scraping new data:

INFO:/Users/bpb/repos/garmin_activity/GarminDB/FitFileProcessor.pyc:/Users/bpb/HealthData/FitFiles/2018_Monitoring//24955272738.fit (2018-09-19 23:59:59) [monitoring_b] message types: ['file_id', 'device_info', 'software', 'monitoring_info', 'monitoring', 'unknown_msg_24', 'event', 'unknown_msg_211']

Here is my attempt to test import it

[bpb@Benjamins-MacBook-Pro-4 ~ 12:54 PM] ~/repos/garmin_activity/GarminDB (master) > make TEST_FIT_FILE=24955272738 test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/bpb/HealthData/FitFiles/Activities/1589795363.fit" --sqlite /Users/bpb/HealthData/DBs

I omit the rest since you can see its not the specified file or path. It jumps out with an IOError: no such file.

The same thing happens no matter which combination of make targets and variable names I pick.

The only obvious thing I see is 2017 is hard-coded into path here: https://github.com/tcgoetz/GarminDB/blob/master/Makefile#L148

tcgoetz commented 6 years ago

I'm not sure which version you have, but if you update, the latest works for me like:

orion-dock:GarminDB_master tgoetz$ make TEST_GC_ID=1654887240 test_import_activities
python import_garmin_activities.py -t1 -e --input_file "/Users/tgoetz/HealthData/FitFiles/Activities/1654887240.fit" --sqlite /Users/tgoetz/HealthData/DBs
INFO:import_garmin_activities.py:Debug: 1
INFO:/Users/tgoetz/Projects/GarminDB_master/FileProcessor.pyc:Reading file: /Users/tgoetz/HealthData/FitFiles/Activities/1654887240.fit

The makefile line uses a ?= so it sets if not set.

tcgoetz commented 6 years ago

After which you get output like:

DEBUG:Fit.File: Message [14]: DataMessage: <UnknownMessageType.unknown_233: 233>: unknown_2([5, 1, 0, 0] ([5, 1, 0, 0])),

which means the file contained an unknown message #233 with one unknown filed #2 consisting of an array of four numbers.

benbowen commented 6 years ago

My bad. I swear I did a git pull a couple days ago, but I just pulled again and a dozen files were updated. I'll reinvistigate now that I'm "more" certain that I'm on current master.

tcgoetz commented 6 years ago

I've been pushing a lot of code over the past few days.

Here's an example of a known message:

DEBUG:Fit.File: Message [3]: DataMessage: <MessageType.device_info: 23>: battery_voltage(None v (65535) [invalid]), device_type(None (255) [invalid]), unknown_8(None (4294967295) [invalid]), unknown_9(None (255) [invalid]), unknown_13(None (65535) [invalid]), unknown_15(None (4294967295) [invalid]), unknown_16(None (4294967295) [invalid]), unknown_17([None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None] ([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])), unknown_30(None (255) [invalid]), sensor_position(None (255) [invalid]), product(GarminProduct.Fenix5_Sapphire (2697)), cum_operating_time(None (4294967295) [invalid]), device_index(0 (0)), ant_transmission_type(None (0) [invalid]), serial_number(3943966595 (3943966595)), product(GarminProduct.Fenix5_Sapphire (2697)), software_version(2.4 (240)), timestamp(2017-04-02 13:45:24 (860089525)), source_type(SourceType.local (5)), unknown_29([None, None, None, None, None, None] ([255, 255, 255, 255, 255, 255])), ant_device_number(None (0) [invalid]), unknown_24(None (0) [invalid]), unknown_23(None (255) [invalid]), manufacturer(Manufacturer.Garmin (1)), hardware_version(None (255) [invalid]), battery_status(None (255) [invalid]), ant_network(None (255) [invalid]),

Note that even in a known message, there are unknown fields and unused fields ('invalid').