tcgoetz / GarminDB

Download and parse data from Garmin Connect or a Garmin watch, FitBit CSV, and MS Health CSV files into and analyze data in Sqlite serverless databases with Jupyter notebooks.
GNU General Public License v2.0
1.11k stars 137 forks source link

No sleep data after 5th Dec 2020 #91

Closed juusotam closed 3 years ago

juusotam commented 3 years ago

I've noticed that no sleep data gets updated to the database after the 5th of December 2020.

In the garmin_summary.db database's days_summaryall sleep times are 00:00:00.00000 from the 4th December 2020 onwards.

I looked through the JSON files on the 4th and 5th of December. There is some change in the sleepWindowConfirmationType variable.

On the 4th it is: "enhanced_confirmed" and after that "enhanced_confirmed_final".

Could there be some change in Garmin's data model to explain this behaviour?

Code block 1:

"dailySleepDTO": { "id": xxxxxx, "userProfilePK": xxxxxx, "calendarDate": "2020-12-04", "sleepTimeSeconds": 31020, "napTimeSeconds": 0, "sleepWindowConfirmed": true, "sleepWindowConfirmationType": "enhanced_confirmed", "sleepStartTimestampGMT": 1607025840000, "sleepEndTimestampGMT": 1607056980000, "sleepStartTimestampLocal": 1607033040000, "sleepEndTimestampLocal": 1607064180000, "autoSleepStartTimestampGMT": 1607025600000, "autoSleepEndTimestampGMT": 1607054700000, "sleepQualityTypePK": null, "sleepResultTypePK": null, "unmeasurableSleepSeconds": 0, "deepSleepSeconds": 2100, "lightSleepSeconds": 17460, "remSleepSeconds": 11460, "awakeSleepSeconds": 120, "deviceRemCapable": true, "retro": false }

Code block 2:

"dailySleepDTO": { "id": xxxxxx, "userProfilePK": xxxxxx, "calendarDate": "2020-12-05", "sleepTimeSeconds": 28440, "napTimeSeconds": 0, "sleepWindowConfirmed": true, "sleepWindowConfirmationType": "enhanced_confirmed_final", "sleepStartTimestampGMT": 1607114640000, "sleepEndTimestampGMT": 1607146740000, "sleepStartTimestampLocal": 1607121840000, "sleepEndTimestampLocal": 1607153940000, "autoSleepStartTimestampGMT": 1607114280000, "autoSleepEndTimestampGMT": 1607146560000, "sleepQualityTypePK": null, "sleepResultTypePK": null, "unmeasurableSleepSeconds": 3120, "deepSleepSeconds": 3000, "lightSleepSeconds": 14700, "remSleepSeconds": 10740, "awakeSleepSeconds": 540, "deviceRemCapable": true, "retro": false }

tcgoetz commented 3 years ago

I duplicated the issue. The underlying issue is that the JSON they give for sleep data isn't parsing:

ERROR:root:Failed to parse /Users/tgoetz/HealthData/Sleep/sleep_2020-12-28.json: Traceback (most recent call last): File "/Users/tgoetz/Projects/GarminDB_develop/utilities/json_file_processor.py", line 104, in _process_files json_data = self.parse_file(file_name) File "/Users/tgoetz/Projects/GarminDB_develop/utilities/json_file_processor.py", line 56, in parse_file return json.load(file, object_hook=parser) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init.py", line 296, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init__.py", line 361, in loads return cls(kw).decode(s) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) File "/Users/tgoetz/Projects/GarminDB_develop/utilities/json_file_processor.py", line 53, in parser entry[conversion_key] = conversion_func(entry_value) File "/Users/tgoetz/Library/Python/3.7/lib/python/site-packages/dateutil/parser/_parser.py", line 1374, in parse return DEFAULTPARSER.parse(timestr, kwargs) File "/Users/tgoetz/Library/Python/3.7/lib/python/site-packages/dateutil/parser/_parser.py", line 646, in parse res, skipped_tokens = self._parse(timestr, **kwargs) File "/Users/tgoetz/Library/Python/3.7/lib/python/site-packages/dateutil/parser/_parser.py", line 725, in _parse l = _timelex.split(timestr) # Splits the timestr into tokens File "/Users/tgoetz/Library/Python/3.7/lib/python/site-packages/dateutil/parser/_parser.py", line 207, in split return list(cls(s)) File "/Users/tgoetz/Library/Python/3.7/lib/python/site-packages/dateutil/parser/_parser.py", line 76, in init '{itype}'.format(itype=instream.class.name__)) TypeError: Parser must be a string or character stream, not int

tcgoetz commented 3 years ago

Committed a fix (fbfd0aa) on the develop branch. Can you checkout the develop branch and see if the fix works for you?