search

tcgoetz / GarminDB

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
GNU General Public License v2.0
1.11k stars 137 forks source link

Mac binary 1.2.4 error when processing Tcx? #45

Closed jentera closed 4 years ago

jentera commented 4 years ago

I got this message after initial creating database from Connect:

Processing activities tcx data
 14% (10 of 71) |###                     | Elapsed Time: 0:00:19 ETA:   0:02:40Traceback (most recent call last):
  File "garmin.py", line 389, in <module>
  File "garmin.py", line 379, in main
  File "garmin.py", line 202, in import_data
  File "import_garmin_activities.py", line 165, in process_files
  File "import_garmin_activities.py", line 112, in __process_file
  File "Tcx/tcx.py", line 195, in read
  File "Tcx/tcx.py", line 177, in update
  File "Tcx/tcx.py", line 148, in __tag_values
  File "Tcx/tcx.py", line 148, in <listcomp>
AttributeError: 'NoneType' object has no attribute 'strip'
[8538] Failed to execute script garmin

Tried different timeframe (changed the start date) on config but it also failed at 14%.

tcgoetz commented 4 years ago

You have an activity in a Tcx file that has altitude entries, but the entries are empty. That isn't something I expected. Making a fix.

tcgoetz commented 4 years ago

Try: https://github.com/tcgoetz/GarminDB/releases/tag/v1.2.5

If you get an error again, please post the new error.

jentera commented 4 years ago

Hi, thanks for looking into this. I'm using Mac v1.2.5 binary now, but still got the same error:

Processing activities tcx data
 14% (10 of 71) |###                     | Elapsed Time: 0:00:35 ETA:   0:03:29Traceback (most recent call last):
  File "garmin.py", line 389, in <module>
  File "garmin.py", line 379, in main
  File "garmin.py", line 202, in import_data
  File "import_garmin_activities.py", line 165, in process_files
  File "import_garmin_activities.py", line 112, in __process_file
  File "Tcx/tcx.py", line 195, in read
  File "Tcx/tcx.py", line 177, in update
  File "Tcx/tcx.py", line 148, in __tag_values
  File "Tcx/tcx.py", line 148, in <listcomp>
AttributeError: 'NoneType' object has no attribute 'strip'
[15692] Failed to execute script garmin

Btw, I also tried to build using ubuntu 18.04 using commit 8d94016 via ssh git clone. After successful make setup, during make create_dbs I got this:

Processing FileType.monitoring_b FIT data from /home/koding/HealthData/FitFiles/Monitoring
100% (117 of 117) |########################| Elapsed Time: 0:00:27 Time:  0:00:27
Processing sleep data
100% (32 of 32) |##########################| Elapsed Time: 0:00:00 Time:  0:00:00
Processing rhr data
100% (32 of 32) |##########################| Elapsed Time: 0:00:00 Time:  0:00:00
Processing activities tcx data
 14% (10 of 71) |###                       | Elapsed Time: 0:00:10 ETA:   0:01:11Traceback (most recent call last):
  File "garmin.py", line 387, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 377, in main
    import_data(debug, latest, weight, monitoring, sleep, rhr, activities, test)
  File "garmin.py", line 202, in import_data
    gtd.process_files(db_params_dict)
  File "/home/koding/GarminDB/import_garmin_activities.py", line 165, in process_files
    self.__process_file(file_name)
  File "/home/koding/GarminDB/import_garmin_activities.py", line 112, in __process_file
    tcx.read(file_name)
  File "/home/koding/GarminDB/Tcx/tcx.py", line 195, in read
    self.update()
  File "/home/koding/GarminDB/Tcx/tcx.py", line 177, in update
    self.altitude_values = self.__tag_values(float, './/ns:AltitudeMeters')
  File "/home/koding/GarminDB/Tcx/tcx.py", line 148, in __tag_values
    return [type_func(value.text.strip()) for value in self.__findall(self.activity, tag_path, namespace) if value is not None]
  File "/home/koding/GarminDB/Tcx/tcx.py", line 148, in <listcomp>
    return [type_func(value.text.strip()) for value in self.__findall(self.activity, tag_path, namespace) if value is not None]
AttributeError: 'NoneType' object has no attribute 'strip'
100% (71 of 71) |##########################| Elapsed Time: 0:00:10 Time:  0:00:10
Command exited with non-zero status 1
/usr/bin/time
Makefile:128: recipe for target 'update_garmin' failed
make: *** [update_garmin] Error 1
tcgoetz commented 4 years ago

Please update your ubuntu tree and try again. You should see commit ccce27d in your Tcx repo. Please reply with success or any new error. Thanks!

jentera commented 4 years ago

Now I got this:

Processing activities tcx data
 14% (10 of 71) |###                       | Elapsed Time: 0:00:23 ETA:   0:02:14Traceback (most recent call last):
  File "garmin.py", line 387, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 377, in main
    import_data(debug, latest, weight, monitoring, sleep, rhr, activities, test)
  File "garmin.py", line 202, in import_data
    gtd.process_files(db_params_dict)
  File "/home/koding/GarminDB/import_garmin_activities.py", line 165, in process_files
    self.__process_file(file_name)
  File "/home/koding/GarminDB/import_garmin_activities.py", line 156, in __process_file
    self.__process_lap(tcx, file_id, lap_number, lap)
  File "/home/koding/GarminDB/import_garmin_activities.py", line 96, in __process_lap
    'start_time'                        : tcx.get_lap_start(lap),
  File "/home/koding/GarminDB/Tcx/tcx.py", line 266, in get_lap_start
    return self.get_point_time(self.get_lap_points(lap)[0])
IndexError: list index out of range
100% (71 of 71) |##########################| Elapsed Time: 0:00:23 Time:  0:00:23
Command exited with non-zero status 1
/usr/bin/time
Makefile:116: recipe for target 'garmin' failed
make: *** [garmin] Error 1

Perhaps there's something wrong with my data in Connect? Now it's lap time?

tcgoetz commented 4 years ago

Tcx files are not usually from Garmin Connect itself. All of the times GC gave me Tcx files was for activities I imported from somewhere else. Tcx files have a lot of variations. Your showing me examples of things I haven't seen before. I'll keep fixing them and hopefully we will have all of your cases covered shortly.

tcgoetz commented 4 years ago

Next round of fixes: https://github.com/tcgoetz/GarminDB/commit/a6ff1bc2668bca0c641204c6de1e5dde128c97d6.

Do a make update and try again.

jentera commented 4 years ago

Hi. Still got another error. I noticed from Connect dashboard that the TCXs processed below doesn't have HR data and the "Elapsed Time" is 0:00. 

Processing daily hydration data
100% (13 of 13) |##########################| Elapsed Time: 0:00:00 Time:  0:00:00
Processing FileType.monitoring_b FIT data from /home/koding/HealthData/FitFiles/Monitoring
100% (217 of 217) |########################| Elapsed Time: 0:01:09 Time:  0:01:09
Processing sleep data
100% (13 of 13) |##########################| Elapsed Time: 0:00:05 Time:  0:00:05
Processing rhr data
100% (13 of 13) |##########################| Elapsed Time: 0:00:00 Time:  0:00:00
Processing activities tcx data
 14% (10 of 71) |###                       | Elapsed Time: 0:00:28 ETA:   0:03:03Exception processing file /home/koding/HealthData/FitFiles/Activities/909019240.tcx: 'NoneType' object has no attribute 'lat_deg'
 15% (11 of 71) |####                      | Elapsed Time: 0:00:28 ETA:   0:01:33Exception processing file /home/koding/HealthData/FitFiles/Activities/909019336.tcx: 'NoneType' object has no attribute 'lat_deg'
 16% (12 of 71) |####                      | Elapsed Time: 0:00:28 ETA:   0:01:32Exception processing file /home/koding/HealthData/FitFiles/Activities/909019388.tcx: 'NoneType' object has no attribute 'lat_deg'
 18% (13 of 71) |####                      | Elapsed Time: 0:00:28 ETA:   0:00:47Exception processing file /home/koding/HealthData/FitFiles/Activities/909019453.tcx: 'NoneType' object has no attribute 'lat_deg'
100% (71 of 71) |##########################| Elapsed Time: 0:03:10 Time:  0:03:10
Processing all activities summary data from /home/koding/HealthData/FitFiles/Activities
100% (188 of 188) |########################| Elapsed Time: 0:00:05 Time:  0:00:05
Processing activities detail data
100% (188 of 188) |########################| Elapsed Time: 0:00:06 Time:  0:00:06
Processing activities extra data
Processing FileType.activity FIT data from /home/koding/HealthData/FitFiles/Activities
N/A% (0 of 116) |                          | Elapsed Time: 0:00:00 ETA:  --:--:--Traceback (most recent call last):
  File "garmin.py", line 387, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 377, in main
    import_data(debug, latest, weight, monitoring, sleep, rhr, activities, test)
  File "garmin.py", line 218, in import_data
    gfd.process_files(db_params_dict)
  File "/home/koding/GarminDB/fit_data.py", line 54, in process_files
    fp.write_file(fit_file)
  File "/home/koding/GarminDB/fit_file_processor.py", line 83, in write_file
    self.__write_message_types(fit_file, fit_file.message_types())
  File "/home/koding/GarminDB/fit_file_processor.py", line 71, in __write_message_types
    self.__write_message_type(fit_file, message_type)
  File "/home/koding/GarminDB/fit_file_processor.py", line 57, in __write_message_type
    function(fit_file, message_type, messages)
  File "/home/koding/GarminDB/fit_file_processor.py", line 46, in __write_generic
    function(fit_file, message_dict)
  File "/home/koding/GarminDB/fit_file_processor.py", line 266, in _write_session_entry
    sub_sport = message_dict['sub_sport']
KeyError: 'sub_sport'
100% (116 of 116) |########################| Elapsed Time: 0:00:02 Time:  0:00:02
Command exited with non-zero status 1
/usr/bin/time
Makefile:116: recipe for target 'garmin' failed
make: *** [garmin] Error 1

Thanks

tcgoetz commented 4 years ago

Couple more fixes. Please try again.

jentera commented 4 years ago

Now I got this:

Processing FileType.monitoring_b FIT data from /home/koding/HealthData/FitFiles/Monitoring
100% (207 of 207) |##############################################################| Elapsed Time: 0:01:34 Time:  0:01:34
Processing sleep data
100% (32 of 32) |################################################################| Elapsed Time: 0:00:09 Time:  0:00:09
Processing rhr data
100% (32 of 32) |################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
Processing activities tcx data
100% (71 of 71) |################################################################| Elapsed Time: 0:02:49 Time:  0:02:49
Processing all activities summary data from /home/koding/HealthData/FitFiles/Activities
100% (188 of 188) |##############################################################| Elapsed Time: 0:00:03 Time:  0:00:03
Processing activities detail data
100% (188 of 188) |##############################################################| Elapsed Time: 0:00:03 Time:  0:00:03
Processing activities extra data
Processing FileType.activity FIT data from /home/koding/HealthData/FitFiles/Activities
N/A% (0 of 116) |                                                                | Elapsed Time: 0:00:00 ETA:  --:--:--Traceback (most recent call last):
  File "garmin.py", line 387, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 377, in main
    import_data(debug, latest, weight, monitoring, sleep, rhr, activities, test)
  File "garmin.py", line 218, in import_data
    gfd.process_files(db_params_dict)
  File "/home/koding/GarminDB/fit_data.py", line 54, in process_files
    fp.write_file(fit_file)
  File "/home/koding/GarminDB/fit_file_processor.py", line 83, in write_file
    self.__write_message_types(fit_file, fit_file.message_types())
  File "/home/koding/GarminDB/fit_file_processor.py", line 71, in __write_message_types
    self.__write_message_type(fit_file, message_type)
  File "/home/koding/GarminDB/fit_file_processor.py", line 57, in __write_message_type
    function(fit_file, message_type, messages)
  File "/home/koding/GarminDB/fit_file_processor.py", line 46, in __write_generic
    function(fit_file, message_dict)
  File "/home/koding/GarminDB/fit_file_processor.py", line 297, in _write_session_entry
    activity.update(self.__choose_sport(current.sport, current.sub_sport, sport, sub_sport))
  File "/home/koding/GarminDB/fit_file_processor.py", line 259, in __choose_sport
    if (sub_sport is None and new_sub_sport is not None) or (not sub_sport.preferred() and new_sub_sport.preferred()):
AttributeError: 'NoneType' object has no attribute 'preferred'
100% (116 of 116) |##############################################################| Elapsed Time: 0:00:02 Time:  0:00:02
Command exited with non-zero status 1
/usr/bin/time
Makefile:116: recipe for target 'garmin' failed
make: *** [garmin] Error 1

I guess the tcx part is fixed now.

tcgoetz commented 4 years ago

Any chance you could share with me one or more of these files? Then I could fix all of the issue instead of one at a time.

/home/koding/HealthData/FitFiles/Activities/909019240.tcx /home/koding/HealthData/FitFiles/Activities/909019336.tcx /home/koding/HealthData/FitFiles/Activities/909019388.tc /home/koding/HealthData/FitFiles/Activities/909019453.tcx

tcgoetz commented 4 years ago

commit e5cf943 fixes the stacktrace above.

tcgoetz commented 4 years ago

Are you still having issues?

jentera commented 4 years ago

Hi, sorry for the late response. I still got this and I'll send you sample of tcx.

100% (188 of 188) |##############################################################| Elapsed Time: 0:00:04 Time:  0:00:04
Processing activities detail data
100% (188 of 188) |##############################################################| Elapsed Time: 0:00:03 Time:  0:00:03
Processing activities extra data
Processing FileType.activity FIT data from /home/koding/HealthData/FitFiles/Activities
N/A% (0 of 116) |                                                                | Elapsed Time: 0:00:00 ETA:  --:--:--Traceback (most recent call last):
  File "garmin.py", line 387, in <module>
    main(sys.argv[1:])
  File "garmin.py", line 377, in main
    import_data(debug, latest, weight, monitoring, sleep, rhr, activities, test)
  File "garmin.py", line 218, in import_data
    gfd.process_files(db_params_dict)
  File "/home/koding/GarminDB/fit_data.py", line 54, in process_files
    fp.write_file(fit_file)
  File "/home/koding/GarminDB/fit_file_processor.py", line 83, in write_file
    self.__write_message_types(fit_file, fit_file.message_types())
  File "/home/koding/GarminDB/fit_file_processor.py", line 71, in __write_message_types
    self.__write_message_type(fit_file, message_type)
  File "/home/koding/GarminDB/fit_file_processor.py", line 57, in __write_message_type
    function(fit_file, message_type, messages)
  File "/home/koding/GarminDB/fit_file_processor.py", line 46, in __write_generic
    function(fit_file, message_dict)
  File "/home/koding/GarminDB/fit_file_processor.py", line 297, in _write_session_entry
    activity.update(self.__choose_sport(current.sport, current.sub_sport, sport, sub_sport))
  File "/home/koding/GarminDB/fit_file_processor.py", line 261, in __choose_sport
    return {'sport' : sport.name, 'sub_sport' : sub_sport.name}
AttributeError: 'NoneType' object has no attribute 'name'
100% (116 of 116) |##############################################################| Elapsed Time: 0:00:04 Time:  0:00:04
Command exited with non-zero status 1
/usr/bin/time
Makefile:116: recipe for target 'garmin' failed
make: *** [garmin] Error 1
jentera commented 4 years ago

Here's a sample of tcx. Thanks.

Pada tanggal Min, 19 Jan 2020 pukul 15.12 Tom Goetz < notifications@github.com> menulis:

Are you still having issues?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tcgoetz/GarminDB/issues/45?email_source=notifications&email_token=AAHUSSXE4ANWEMSCEMAUXZLQ6QDN7A5CNFSM4KCZSVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKLXEY#issuecomment-575978387, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHUSSVUT2ZCJZWXKK7WTMLQ6QDN7ANCNFSM4KCZSVVA .

tcgoetz commented 4 years ago

The above commits should resolve this error:

AttributeError: 'NoneType' object has no attribute 'name'

jentera commented 4 years ago

Yes, it worked! Thank you so much.