search

cagnulein / qdomyos-zwift

Zwift bridge for smart treadmills and bike/cyclette
https://www.qzfitness.com/
GNU General Public License v3.0
369 stars 109 forks source link

QZ FIT file start timestamp inconsistent with Zwift #1311

Open victorypoint opened 1 year ago

victorypoint commented 1 year ago

@cagnulein, now that we have a consistent way to synchronize that start of activities between Zwift and QZ (ensure speed is 0 then increase to start FIT file recording at the same time), I'm finding a strange inconsistency with the start timestamp in the QZ FIT file. This is affecting the ability to accurately merge data between the two FIT files (e.g. latitude, longitude, elevation, power, etc).

For example, here's results of a recent test using QZ fake treadmill mode:

I've retested this several times and always arrive at a strange delayed QZ starting timestamp. Any help resolving this is greatly appreciated.

FIT files attached. test4.zip

victorypoint commented 1 year ago

A few more notes. Using https://www.fitfileviewer.com/, the Session and Event tables of QZ FIT file show a start timestamp of 2:14:03pm. That could possibly indicate there is missing data from 2:14:03 - 2:17:01pm in the QZ file.

cagnulein commented 1 year ago

seems a bug in the garmin fit library. If you look in the fitfileviewer website, the session record has a good timestamp, that's because, instead of using the date object from the fit library, i'm calculating it myself. also the lap timestamp is wrong, and it's using the fit object too...so i have to use my method only.

Question: if I create a windows version with the patch, can you help me doing a test and verify that it's all good?

cagnulein commented 1 year ago

@victorypoint when it's ready you will see a new windows version here https://github.com/cagnulein/qdomyos-zwift/actions/runs/4267692357 (bottom of the page) it will be ready in 30 minutes

victorypoint commented 1 year ago

@cagnulein, thank-you! Very excited to test this out. That's interesting about the timestamp bug in the Garmin FIT SDK. I'm curious if you're using the latest 21.105 SDK, Feb. 2, 2023? - https://developer.garmin.com/fit/download/

cagnulein commented 1 year ago

no I updated the last time some months ago. I usually don't use never the last version due to possible new bugs. When I find something stable I usually stick with it ;)

Roberto Viola Software engineer and open source enthusiast http://robertoviola.cloud

Il giorno sab 25 feb 2023 alle ore 02:56 Al Udell @.***> ha scritto:

@cagnulein https://github.com/cagnulein, thank-you! Very excited to test this out. That's interesting about the timestamp bug in the Garmin FIT SDK. I'm curious if you're using the latest 21.105 SDK, Feb. 2, 2023? - https://developer.garmin.com/fit/download/

— Reply to this email directly, view it on GitHub https://github.com/cagnulein/qdomyos-zwift/issues/1311#issuecomment-1444905535, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALYWDNKGNPEJ7BHASC3TLWZFRDFANCNFSM6AAAAAAVHIEN2Y . You are receiving this because you were mentioned.Message ID: @.***>

cagnulein commented 1 year ago

@victorypoint ready https://github.com/cagnulein/qdomyos-zwift/suites/11196536723/artifacts/572412013

victorypoint commented 1 year ago

@victorypoint ready https://github.com/cagnulein/qdomyos-zwift/suites/11196536723/artifacts/572412013

Thank-you. I started doing some testing. Go to bed! 😴🛌

victorypoint commented 1 year ago

@cagnulein, using new Windows QZ build v2.12.73, I repeated my test as before using QZ fake treadmill mode::

Here are detailed timestamps per FIT table:

FIT files attached. test1.zip

cagnulein commented 1 year ago

sorry @victorypoint I found now the original issue, it's not fault of the fit library, it's just my fault :( New build here in the bottom https://github.com/cagnulein/qdomyos-zwift/actions/runs/4268708107

cagnulein commented 1 year ago

@victorypoint in case you missed, new artifact is ready

victorypoint commented 1 year ago

@cagnulein, this update is great! I retested as before. Here's the results:

Looking at the FIT tables using https://www.fitfileviewer.com/, I noticed both FIT files have consistent data structure, however the QZ Event table has no stop timestamp record whereas the Zwift table does. Is this important?

I will continue to do more testing.

cagnulein commented 1 year ago

ok let me know when i can merge this!

Il giorno sab 25 feb 2023 alle 23:41 Al Udell @.***> ha scritto:

@cagnulein https://github.com/cagnulein, this update is great! I retested as before. Here's the results:

  • Launch QZ in paused state - 2:38pm
  • Launch Zwift and begin a manual run - 2:42pm - avatar stands still waiting for speed
  • Hit QZ Start button - 2:43pm - avatar stands still waiting for speed
  • Hit QZ preset speed button 10kph - 2:45pm - avatar begins to move
  • Hit QZ Stop button, end Zwift run, exit both QZ and Zwift - 2:55pm
  • Examine Zwift and QZ FIT files:
    • Zwift starting timestamp - 2:45:07pm - this appears to be very close when QZ speed preset button was pressed
    • QZ starting timestamp - 2:45:04pm - this also appears to be very close QZ speed preset button was pressed

Here's the original Zwift FIT file elevation graph overlayed with the elevation corrected FIT file:

[image: image] https://user-images.githubusercontent.com/63697253/221381711-65686043-521e-492f-aecb-06cb22bb6fa4.png

I will continue to do more testing.

— Reply to this email directly, view it on GitHub https://github.com/cagnulein/qdomyos-zwift/issues/1311#issuecomment-1445220338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALYWAR7LSB4PW6S2BOBYLWZKDBNANCNFSM6AAAAAAVHIEN2Y . You are receiving this because you were mentioned.Message ID: @.***>

-- Roberto Viola Software engineer and open source enthusiast http://robertoviola.cloud

victorypoint commented 1 year ago

@cagnulein, my next test discovered a problem with how QZ calculates elevation gain. I'll post the issue here but let me know if you want a new ticket for this.

For this test, I wanted to compare how QZ calculated elevation compares with the terrain elevation for a given course. To get a terrain elevation graph, I used bike mode and mapped the entire 23K Volcano Climb course in a FIT file. Next, I switched to run mode (using the bike pairing meetup trick) and repeated the course using auto-inclination, then merged the QZ elevation into the resulting Zwift FIT file. Here's the 2 graphs overlayed. You can see the Zwift bike elevation in blue and QZ run elevation in green. From what I can tell, the ascent elevations are very close, however QZ appears to be shallowing the decent.

volcano climb - graph comparison

Note that I used fake treadmill for this test with 'Zwift inclination gain' = 2. For comparison, here is an elevation graph of Zwift bike elevation overlayed with QZ bike calculated elevation.

GPXSee 2023-02-27 10_31_36 AM

So, it appears that QZ is treating decent (negative elevation) with a different calculation than ascent (positive elevation).

cagnulein commented 1 year ago

yes because zwift send the negative inclination to half :) you have to enable the zwift negative incliantion double setting. There are some youtube videos about this zwift issue too :)

victorypoint commented 1 year ago

@cagnulein, I repeated this test using the longer Mega Pretzel course just to confirm it wasn't isolated to a single course. Again I'm finding that QZ is treating decent (negative elevation) with a different calculation than ascent (positive elevation). Here's the 2 graphs overlayed - Zwift bike elevation in blue and QZ run elevation in green. Again, I'm using fake treadmill with 'Zwift inclination gain' = 2.

uber pretzel - graph comparison

Also, here is the elevation graph of Zwift bike elevation overlayed with QZ bike calculated elevation.

GPXSee 2023-02-26 2_27_00 PM

cagnulein commented 1 year ago

check my above comment @victorypoint

victorypoint commented 1 year ago

yes because zwift send the negative inclination to half :) you have to enable the zwift negative incliantion double setting. There are some youtube videos about this zwift issue too :)

Thanks! .. desperately poking around in QZ looking for this setting now...

victorypoint commented 1 year ago

yes because zwift send the negative inclination to half :) you have to enable the zwift negative incliantion double setting. There are some youtube videos about this zwift issue too :)

Sorry, I can't find this setting. Why exactly is Zwift sending negative inclination to half and positive inclination to full?

cagnulein commented 1 year ago

bike options, double negative inclination. zwift does this because, IMHO, the bikes with negative inclination will be uncontrollable (because you will have super high cadence during downhill)

victorypoint commented 1 year ago

@cagnulein, found it under Bike options - 'Double Negative Inclination'. So it's a Zwift bug?

cagnulein commented 1 year ago

yes confirmed :)

victorypoint commented 1 year ago

yes confirmed :)

Ok thanks. Will retest.

victorypoint commented 1 year ago

@cagnulein, so with QZ 'Double Negative Inclination' turned on this time, unfortunately I'm getting the same elevation graphs as before. I'm thinking maybe there's another setting that needs to be set in QZ. I'm using fake treadmill with 'Zwift inclination gain' = 2 as before with Wahoo Treadmill device.

GPXSee 2023-02-27 10_33_45 PM

Logs and FIT files attached. bike vs run mode.zip

cagnulein commented 1 year ago

oh that's strange, let me check this!

Il giorno mar 28 feb 2023 alle 03:40 Al Udell @.***> ha scritto:

@cagnulein https://github.com/cagnulein, so with Zwift 'Double Negative Inclination' turned on this time, unfortunately I'm getting the same elevation graphs as before. I'm thinking maybe there's another setting that needs to be set in QZ. I'm using fake treadmill with 'Zwift inclination gain' = 2 as before with Wahoo Treadmill device.

[image: GPXSee 2023-02-27 10_33_45 PM] https://user-images.githubusercontent.com/63697253/221738963-b2edf4df-9b6d-4a64-9672-3ae1d652bac4.png

Logs and FIT files attached. bike vs run mode.zip https://github.com/cagnulein/qdomyos-zwift/files/10846032/bike.vs.run.mode.zip

— Reply to this email directly, view it on GitHub https://github.com/cagnulein/qdomyos-zwift/issues/1311#issuecomment-1447481782, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALYWGMA5EGH7774UW7WV3WZVQSZANCNFSM6AAAAAAVHIEN2Y . You are receiving this because you were mentioned.Message ID: @.***>

-- Roberto Viola Software engineer and open source enthusiast http://robertoviola.cloud

cagnulein commented 1 year ago

@victorypoint i'm an idiot, i'm applying the zwift negative inclination only to the bikes. That's why you're seeing this. Ok so now I'm doing 2 modifications

  1. using this settings for all the devices
  2. moving the setting to the advanced settings
cagnulein commented 1 year ago

done @victorypoint you can check this new modification here https://github.com/cagnulein/qdomyos-zwift/actions/runs/4290568258 (always on the bottom for the artifact)

victorypoint commented 1 year ago

done @victorypoint you can check this new modification here https://github.com/cagnulein/qdomyos-zwift/actions/runs/4290568258 (always on the bottom for the artifact)

Ok awesome. I will test this morning. One thing I noticed yesterday which I should have posted is the erg grade was different than the transmitted incline. Not sure if they are supposed to match? For example:

calculated erg grade 3.99" changeInclination 5.32 true 1 0 calculated erg grade 4.08" changeInclination 5.44 true 1 0 calculated erg grade 4.095" changeInclination 5.46 true 1 0 calculated erg grade 4.2" changeInclination 5.6 true 1 0

cagnulein commented 1 year ago

no they shouldn't there is a radian degrees conversion in the middle

victorypoint commented 1 year ago

@cagnulein, I just completed a bike test using the new QZ build, same course and settings. Here's the elevation graph showing both Zwift elevation in blue and QZ calculated elevation in green. For bike these graphs should match close?

GPXSee 2023-02-28 11_48_30 AM

cagnulein commented 1 year ago

yes do you have the debug log? I guess I'm missing something

cagnulein commented 1 year ago

hah wait, for the bike the issue is the speed, the speed can't match to zwift, did you check? because zwift calculates the speed based on power @victorypoint

victorypoint commented 1 year ago

hah wait, for the bike the issue is the speed, the speed can't match to zwift, did you check? because zwift calculates the speed based on power @victorypoint

Ok, that sounds like the problem then. I'm just setting speed to 10kph in QZ which translates to around 35-38kph for bike

cagnulein commented 1 year ago

yes I guess the only test that matters it's with the treadmill mode. Sorry if I didn't raise this first, but I just thought about this right now!

victorypoint commented 1 year ago

for the bike the issue is the speed, the speed can't match to zwift, did you check? because zwift calculates the speed based on power

No worries. I'll test this later after my ZLDR runs.

victorypoint commented 1 year ago

@cagnulein, this is looking better. Here's Zwift vs QZ run elevation on same course..

GPXSee 2023-02-28 3_49_36 PM

And Zwift bike elevation vs QZ run elevation, same course. The graph profiles look almost identical but I don't understand why they're offset about 20 metres?

Videos 2023-02-28 3_50_56 PM

cagnulein commented 1 year ago

green is always qz in the second chart? if so i don't understand either the offset. do you have the debug log?

victorypoint commented 1 year ago

@cagnulein, I think I understand... I started at elevation

green is always qz in the second chart? if so i don't understand either the offset. do you have the debug log?

Yes, QZ is green. Hang on... I think I know why.. retesting...

cagnulein commented 1 year ago

ok meanwhile i'm going to bed :D

Il giorno mar 28 feb 2023 alle 21:37 Al Udell @.***> ha scritto:

@cagnulein https://github.com/cagnulein, I think I understand... I started at elevation

green is always qz in the second chart? if so i don't understand either the offset. do you have the debug log?

Yes, QZ is green. Hang on... I think I know why.. retesting...

— Reply to this email directly, view it on GitHub https://github.com/cagnulein/qdomyos-zwift/issues/1311#issuecomment-1448881445, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALYWBXLQLZV3XSP4PUL2LWZZOZFANCNFSM6AAAAAAVHIEN2Y . You are receiving this because you were mentioned.Message ID: @.***>

-- Roberto Viola Software engineer and open source enthusiast http://robertoviola.cloud

victorypoint commented 1 year ago

@cagnulein, I retested... so it looks like the vertical offset between the 2 elevation profiles is due to Zwift runner having different elevation at the beginning of the run (FIT file). So QZ calculates the entire course elevation relative to the starting elevation. In this case, the difference is:

But also note when comparing the 2 graphs that the horizontal start is not exactly the same. I think this is due to the difference in speed between bike and run tests. But this also adds a bit more elevation difference between the 2 profiles.

bike vs run

So it seems that when running with auto-inclination, the QZ elevation profile will always be relative to the QZ starting elevation. Ideally, this starting elevation should be the terrain elevation for that location but we have no way of getting this information (Zwift doesn't send it, just % grade).

Is there's no way to reset the QZ start elevation unless QZ is stopped and started (setting everything back to 0)? So somehow the user would need to stop/start QZ at the beginning of each run in order to set elevation to 0. But this sets start elevation to 0 which is likely not always the actual terrain elevation.

cagnulein commented 1 year ago

Is there's no way to reset the QZ start elevation unless QZ is stopped and started (setting everything back to 0)? So somehow the user would need to stop/start QZ at the beginning of each run in order to set elevation to 0. But this sets start elevation to 0 which is likely not always the actual terrain elevation.

Yes there is no way to sync them precisely, you should reset elevation to qz (pressing stop) and then start from a Zwift elevation of 0 (very unlikely)

victorypoint commented 1 year ago

@cagnulein, I'm just throwing out some ideas on this FIT file sync issue. Let me know your thoughts on how we could simplify or automate the process.

Here's an example of QZ files recorded for a Zwift activity. Filenames and timestamps appear to be generated at various times as the code executes:

qz files

cagnulein commented 1 year ago
  • How can we trigger QZ actions (Stop, Start) automatically to coincide with Zwift activities? In my case, QZ is always running in the background of full-screen Zwift, so accessing the buttons to stop/start QZ is not easy.

maybe with intent you can automate it with tasker or similar? or maybe we can put a setting that when the speed is greater than 0 it start a new activity and it stop when the speed is equal to 0?

  • Once the 2 FIT files are complete, then merging them currently requires a command-line process. The problem is Zwift and QZ FIT filenames and timestamps don't match, making it difficult to match up the files for merging. For example:

since it's a terminal application the both files could be passed as an arguments, don't?

victorypoint commented 1 year ago

maybe with intent you can automate it with tasker or similar? or maybe we can put a setting that when the speed is greater than 0 it start a new activity and it stop when the speed is equal to 0?

That's a great idea but I can foresee problems as users would be doing a bit of running before an event starts for example (avatar running on virtual treadmill). I like the idea of a tasker or AHK hotkey or bluetooth button press. Will test it out.

since it's a terminal application the both files could be passed as an arguments, don't?

I have an idea to test...

victorypoint commented 1 year ago

@cagnulein, I adapted these Java utilities to merge Zwift FIT file with QZ FIT file to add Zwift location data to QZ. They both use the Garmin FIT SDK.

  1. Fork of nickrodie/Zmerge repo - https://github.com/victorypoint/Zmerge

    • Zmerge is a Java app with GUI. You select QZ FIT file and Zwift FIT file as input and specify location of merged FIT output

  2. Fork of nickrodie/garmin-zwift-merge - https://github.com/victorypoint/qz-zwift-merge

    • GarminZwiftMerge is a Java app for command-line and batch files. Syntax is "GarminZwiftMerge -g qz.fit -z zwift.fit". Output defaults to the name of the Zwift file with .merged.fit

Is it worth incorporating this code into QZ or just run independently?

cagnulein commented 1 year ago

Wow great work! I feel difficult to incorporate this in QZ directly, It will be better to keep it as a separate repository as I'm doing also for the Companions. Do you agree?

victorypoint commented 1 year ago

Wow great work! I feel difficult to incorporate this in QZ directly, It will be better to keep it as a separate repository as I'm doing also for the Companions. Do you agree?

Sure. No worries. It will be easier now at least to set up a batch merge process.

victorypoint commented 1 year ago

... or maybe we can put a setting that when the speed is greater than 0 it start a new activity and it stop when the speed is equal to 0?

@cagnulein, I think this may be worth testing. I'm liking the idea more and more.

cagnulein commented 1 year ago

the only thing is that how handle the pause event :) btw can i merge the main modification of this ticket ? i guess it's working and also it's safe

Il giorno gio 2 mar 2023 alle 20:00 Al Udell @.***> ha scritto:

... or maybe we can put a setting that when the speed is greater than 0 it start a new activity and it stop when the speed is equal to 0?

@cagnulein https://github.com/cagnulein, I think this may be worth testing. I'm liking the idea more and more.

— Reply to this email directly, view it on GitHub https://github.com/cagnulein/qdomyos-zwift/issues/1311#issuecomment-1452404378, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAALYWAXXJBHP6FIFJUVU4DW2DU4ZANCNFSM6AAAAAAVHIEN2Y . You are receiving this because you were mentioned.Message ID: @.***>

-- Roberto Viola Software engineer and open source enthusiast http://robertoviola.cloud

victorypoint commented 1 year ago

the only thing is that how handle the pause event :)

Is this what you mean? We will likely need Settings-Treadmill Options-"Pause when App Starts" turned on. First Zwift activity, we press Start button start then Stop button when done. Then reset QZ for next activity by pressing Back and then Pause. Or we leave "Pause when App Starts" turned off. Then we press Pause then Start at beginning of activity.

btw can i merge the main modification of this ticket ? i guess it's working and also it's safe

Yes, it's working well!

cagnulein commented 1 year ago

Yes, it's working well!

merged!

We will likely need Settings-Treadmill Options-"Pause when App Starts" turned on. First Zwift activity, we press Start button start then Stop button when done. Then reset QZ for next activity by pressing Back and then Pause. Or we leave "Pause when App Starts" turned off. Then we press Pause then Start at beginning of activity.

yeah I mean I was looking to find a way to achieve this without adding any new settings. If I'm right, QZ elapsed timer is not counting if the speed is == 0, if this is true the start trigger is ok for the fit file. For the stop instead it's tricky because there is no automatic way to stop it. But I was thinking: is it really necesseary? I mean since the Zwift activity has an end, we can simple discard all the remaining qz activity from the zwift one (the qz activity will be probably longer than the zwift one, but we can simply cut the remaining off, isn't it?)