search

CTHRU / Hitrava

Convert your Huawei Health sport activities and import them in Strava.
Other
352 stars 34 forks source link

[BUG] Incorrect output dates (several years between data points) #64

Closed szakeetm closed 3 years ago

szakeetm commented 3 years ago

Describe the bug

My output TCX file contains dates in 2107 or later, and each GPS data point differs by several months or years (in reality logged each 5 seconds). Strava import understandably rejects these files. Problem is the same with web and local (Python) version. My computer has US English region setting, same problem with MacOS or Windows. I guess the date format is incorrectly parsed.

Some files parse correctly, some not, in any case, there are no errors thrown by the script (see attached log).

Logs Files Attaching input JSON and an example output TCX for a run that contains incorrect dates.

Additional context What I did: 1) Downloaded health data from Huawei in encrypted zip 2) Clone GIT repo and added 7za.exe 3) Ran python.exe .\Hitrava.py --zip ..\HUAWEI_HEALTH_20210426165642.zip --password *** --json_export

Input (all runs): [link removed] Hitrava.py execution log: [link removed] Output from script (example run from 22 sep 2020): [link removed] Output from web converter for the same run: [link removed]

CTHRU commented 3 years ago

Thanks for your detailed report. I can reproduce the wrong result here. I'll try to figure out what's going on.

CTHRU commented 3 years ago

Well, the good news is, I see what's going on. The bad news, I'm not sure I can do something about it.

The timestamps in the original Huawei HiTrack data from your example from 22 sep 2020 are unfortunately all over the place.

This explains your correct observation that timestamps from records are years apart.

Maybe I can investigate further if you could answer a few questions I have at this time:

  1. Do you have this behavior with all your activities or just this one?

  2. I'm not excluding maybe some changes were made to the export format, however the location's timestamp data seems too random as if only a part from the original timestamp data was left. Now, you sent me a saved export of that 1 activity. Basically this is a dump of the raw data, BUT after some superfluous data was stripped. It could be that the time format changed and now gets stripped as well. You could check this by opening the original extracted json file and search the number 1600769064000 in the file. You should find data like below:

"startTime":1600769064000,"attribute":"HW_EXT_TRACK_DETAIL@istp=lbs;k=0;lat=(some number) ;lon=(some number);alt=(some number);t=43466;

Now look for the number after the t= in bold above. Does it say 43466 or another value (with dots or commas in it)?

Thanks for taking your time to check.

szakeetm commented 3 years ago

Thank you for your effort and also for your quick reply. As you said, the culprit is the export format from Huawei. Between us, I see 0 chance that they would fix the feature, so I'm checking if there's maybe a local fix.

Up to my knowledge, I attached the original motion path detail data.json file (reattaching again: [link removed]), which contains all activities, and in addition, two converted activities for the example run on 22 sep 2020. (one for local and one for web output).

I have made a search in the original JSON file, and found exactly one line with 1600769064000 startdate, and indeed it is t=43466

Subsequent lines have the following epoch and relative datestamps:

"startTime":1601485795000 [ ... ] t=69023 //date: Wednesday, September 30, 2020 7:09:55 PM
lat=46.169660;lon=6.020127;alt=433.591077;t=69028;
lat=46.169753;lon=6.020060;alt=433.646951;t=69032;
lat=46.169836;lon=6.020005;alt=433.802413;t=69036;
lat=46.169923;lon=6.019955;alt=433.134482;t=69040;
lat=46.169819;lon=6.020032;alt=435.087032;t=69048;
[...]
"startTime":1602090349000, [...] lat=46.318247;lon=6.175500;alt=418.422475;t=68755 //date: Wednesday, October 7, 2020 7:05:49 PM
"startTime":1602151551000, [...] lat=46.235917;lon=6.039081;alt=453.474454;t=43559 //date: Thursday, October 8, 2020 12:05:51 PM

Using the epoch to time converter, for me it seems that Huawei's format is:

This actually works for any run in the original JSON file.

If I were you, I'd maybe keep this bug open, and if other users report the same format, we could figure out if the script could handle these time formats as well. (Date from epoch timestamp, time from time in elapsed seconds of the day)

CTHRU commented 3 years ago

Brilliant find there! Thank you for looking into that detail and finding the 'time in seconds during the day'. I think I will just implement it, since it appears in all your activities and is consistent. I wonder though what will happen if one will do an activity around midnight though. We'll find out soon, I guess.