search

pcolby / bipolar

Free your data from Polar FlowSync
GNU General Public License v3.0
115 stars 16 forks source link

Suggestion: Allow correction of left-right-balance data #81

Closed remifreiwald closed 6 years ago

remifreiwald commented 6 years ago

Recently, a colleague of mine and I were discussing the data Bipolar is generating regarding left-right-balance.

The numbers generated for left-right-balance should lie between 0 and 100. But there are values outside of this range. Probably not the fault of Bipolar; we think the data from V800 and its sensors is already wrong.

Our suggestion: Values outside of this range should be corrected to 50.

Further thoughts:

Since this would alter the data, it could be practical to have a checkbox which allows to turn this feature on/off. l-r balance

remifreiwald commented 6 years ago

Of course, one could argue that a value of 50 is just as wrong as a value of 130, if it should have been 90 at this point... Both are 40 points from the desired value...

Maybe it would be better to correct negative outliers to 0 and positive outliers to 100?

But that would not improve the teal-colored valleys in the screenshot.

pcolby commented 6 years ago

Thanks @remifreiwald, some good suggestions there. Bipolar internally does a few things to determine LR balance, and its quite possible that the code is not quite right. I'll definitely take a look.

Just to confirm, are we talking HRM? (I'm not seeing LR balance values in the GPX/TCX code, but I could just be missing it)

Thanks.

remifreiwald commented 6 years ago

Hey, yes, we only checked HRM files.

pcolby commented 6 years ago

Hi @remifreiwald, I've made a few tweaks to the way LR balance is calculated already... just to make the calculation more correct when data samples are missing (ie the sensor was temporarily offline). But this doesn't do anything (yet) to validate / correct for erroneous extreme values.

I'd be very interested in seeing how the data looks after the most recent code changes. Can you test using the current master? (takes some advanced knowledge, so not surprised if you can't) Otherwise, can you provide some test data set that exhibits this issue (maybe I already have some?) and explain to me how to get the view you included above so I can play with it locally?

Thanks.

remifreiwald commented 6 years ago

Hi! I will send you new data which contains different balancing values.

In the meantime you can work with the last data set which included power and laps. There are no extreme values outside of the normal range (0 to 100), but there are still segments with either 0 or 100.

To get the same view in Polar ProTrainer 5:

curves menu

pcolby commented 6 years ago

Unfortunately I don't access to a Window machine at the moment, so can't run PPT5 currently.

I you have a moment, you could have quick look at the test data pre/post this change by comparing, for example training-sessions-267510822.hrm@1bd0eda to training-sessions-267510822.hrm@a6ef773.

Thanks.

remifreiwald commented 6 years ago

Unfortunately the exercise contains only very "limited" balance data. The values are either 50 or 0.

In the screenshots you can see, that all 0 values were converted to 50. And we can say, that PT5 is able to open the files. So, at least that works.

My colleague will send me a suitable exercise file (probably monday). And I will try to compile the code myself. Just never worked with Qt before, only Java and PHP stuff...

PRE pre

POST post

remifreiwald commented 6 years ago

Hey, I got new data, which contains more realistic balance values. The only thing missing is values outside of the range 0-100.

Attached you will find the data and the CSV/TCX exports from Flow.

I couldn't compile Bipolar, yet, because the Qt installation fails due to a time-out, have to check this again...

balance-testdata.zip

pcolby commented 6 years ago

Thanks @remifreiwald, here's a copy the resulting HRM file before and after the changes I made above.

training-sessions-1988256040.zip

Looks like its all good now. But as you say, there's no extreme values outside the 0-100 range anyway. I think I'll keep this open until we get some more data that shows the problem (happy to fix it, just want to fix as close to the source of the bad data as possible 😄 )

Thanks again!

remifreiwald commented 6 years ago

I checked the files in ProTrainer 5 and found out they actually contain outliers:

No. 1 Time: 0:02:32 Power: 29 Watts Balance: L255 & -155R

No. 2 Time: 0:02:37 Power: 0 Watts Balance: L255 & -155R

No. 3 Time: 0:02:55 Power: 0 Watts Balance: 176 & -76R

I don't know if this is coming from the raw V800 data or if this is interpretated by Bipolar, but I guess, if we can fix this, everything should be fine :)

pcolby commented 6 years ago

Ah, yep, can see the outliers in the data now. Will take a look :)

pcolby commented 6 years ago

BTW, Bipolar already enforces a technical limit (imposed by the HRM format itself) for the balance to be between 0 and 255 (inclusive):

stream << '\t' << qMax(qMin(leftBalance, 255), 0);

Which is why you don't see any negatives, but I've updated the code (just locally for now) to warn on negatives as well (was already warning on >100%), so now I get:

QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of 355 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of 335 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -44 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -492 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -492 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -176 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of 176 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -387 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -226 % is outside the range 0..100
QWARN  : TestTrainingSession::toHRM(training-sessions-1988256040) leftBalance of -226 % is outside the range 0..100

There you can see the three extreme positives you noted in your comment above, but there's actually ~7 negatives that were already being silently capped too.

Anyway, will look into the source of the errors next, and code around accordingly.

pcolby commented 6 years ago

Hi @remifreiwald, just checking, this isn't a fixie is it? (not expect it is). Just asking 'cause there's one sample where the left and right power samples are both negative, which is technically not possible (well, not this magnitude) on a coaster system. ie proves the data is wrong (at least for that one sample).

pcolby commented 6 years ago

Almost done fixing this one, but just a quick note to point out that @remifreiwald's most recent data set includes a enough negativity in one sample, that the total power for that sample in negative, which results in the generated TCX files being invalid, since Garmin ActivityExtesion schema requires power to be a positive integer.

So I'll fix that up as part of this change too :)

pcolby commented 6 years ago

Ok @remifreiwald, I've updated the code to filter out negative power samples, and restrict the LR balance to between 0 and 100 (c554552909ca1b8292509ffa735bf96ce94cca7b). This is in addition to the earlier change to use LR balance 50 when total power is 0, and both L and R match.

Have a look at the resulting HRM file, and let me know what you think: training-sessions-1988256040.hrm

Thanks.

remifreiwald commented 6 years ago

Hi @pcolby! Sorry, for the late answer...

The test gear used wasn't a fixie. It was a single gear, but freewheeling was possible. We collected the data on a Wahoo Kickr (https://eu.wahoofitness.com/devices/bike-trainers) and took the power from Polar Kéo Power Pedals (https://support.polar.com/en/support/keo_power_bluetooth_smart).

My colleague, who was riding the bike, thinks it is possible, that he may have pulled on both pedals right at the moment of quickly standing up, to sprint for a couple of seconds.

I opened the HRM file in ProTrainer 5 and everything looks great!

I found only one point where Power is 0 and LR balance is: L100 - 0R It's at 0:02:53, but this is a minor thing and ProTrainer 5 ignores it anyway.

I think it's much better now!

pcolby commented 6 years ago

Thanks @remifreiwald :)

I found only one point where Power is 0 and LR balance is: L100 - 0R

Good pickup. I can see the issue - the "equal to use 50" logic was comparing the pre-force-positive values... easy fix, coming up :)

pcolby commented 6 years ago

I just pulled the trigger on the v0.5.7 release... it'll take several hours before Travis CI releases the dmg and around 24 hours (!!) before AppVeyor releases the exe to the GitHub release page (if all goes well).

Edit: had to cancel those builds (it was going to try to release a non-existent OSX build).

New builds are now:

pcolby commented 6 years ago

Builds finished. Latest installers available at https://github.com/pcolby/bipolar/releases/tag/v0.5.7 :)