Di2 D-Fly gear change data from Garmin Edge 1000

[takenokotaketake]

Thank you good script.Sir. very very good. I have GARMIN EDGE1000 and shimano Di2 D-fly. But I cant get Di2 gear change and gear info data on GARMIN EDGE 1000 RAW fit file. Is this script cant yet? I trust this script can it, But I dont understand pick up gear data method, thanks. If you need gear data in fit file. please message me. I gonna send it. RGDS.

[adriangibbons]

Hi @takenokotaketake

I haven't had access to a file with this data before. If you post a link to an example file (e.g. Dropbox) or email it to [...] then I will take a look and incorporate.

It may be that the class is not listening for the right FIT messages from the standard.

Cheers, Adrian.

[takenokotaketake]

gmail send now, RGDS.

[adriangibbons]

Hi @takenokotaketake - I've had a look at the file that you've sent through and the good news is that I have found the gear change event information in the file you sent me and have successfully used the data in a quick test.

I've started work on a gearChanges() method that will return data similar to that available on di2stats.com

I'll give you an update when the method is implemented on GitHub - hopefully later this week!

Cheers, Adrian.

[takenokotaketake]

Thanks very much Sir.

[adriangibbons]

Hey, I've quickly hacked something together. I don't have much time for the rest of the week, but if you want to have a play with gearChanges() it's in the Di2-D-Fly-gearChange branch.

See https://github.com/adriangibbons/php-fit-file-analysis/commit/f7a48e2f6a7981909d878ea7beccba09c53cc94d

Comes with the following caveats:

• code is ugly and names of variables etc are probably not ideal
• it is likely to change at any time
• the output does not match your link to di2stats.com

Keen for thoughts on:

• whether the data in the arrays are useful?
• should we have two-dimensional array of [front][rear]?
• should we have 't' e.g. 50t or just 50?

Will spend more time on the weekend looking at this!

Cheers, Adrian.

[takenokotaketake]

Thank you Sir. I checking now.

I think so... weather is not important I need front and rear 2 arrays (i gonna setting front rear mysql columm) t is not need, because I caliculate gear ratio, then t is attached cannot caliculate or regexp using.

At some question

1. battery status is all 255(look like invalid_value?), it is really true? I gonna wrote battery spend graph, I think 'battery_status' => [1 => 'new', 2 => 'good', 3 => 'ok', 4 => 'low', 5 => 'critical', 7 => 'unknown'],

Y=battery status, $pFFA->data_mesgs[device_info][battery_status] X= timestamp $pFFA->data_mesgs[device_info][timestamp]

2 session data was very suspicion. x "avg_speed": 5.619, Garmin_connect is 20.2 x "max_speed": 13.52, Garmin_connect is 48.7 x "total_ascent": 366, Garmin_connect is 1008

gear infomation is where? I think this

gear_info:[ front:[ [1]=>50, [2]=>34 }, rear:[ [1]=>11, [2]=>13, [3]=>15, [4]=>17, [5]=>19, [6]=>22, [7]=>25, .............................. Mr. adriangibbons, I just wanna tell you it was a real pleasure watching you work. cheers.

[adriangibbons]

1. Battery status - I will need to look into this but yes it would seem an invalid value if 255.
2. Session average speed - I suspect it is in m/s, the raw value from the file. This is a bug as I have overlooked setting the units for activity/session/lap data with setUnits(). I'll open a separate issue for this! Good catch!

Hey, I'm just happy to be coding if people are finding it useful!

Adrian.

[adriangibbons]

Yes, 5.619m/s is 20.2km/h and 13.52 is 48.7

What units is altitude for Garmin Connect? m or feet?

It could be that Garmin Connect is overriding the altitude data recorded by the Garmin with elevation data based on latitude and longitude.

[takenokotaketake]

Garmin connect is All metric system,Sir! ave = km/h

so anyway! Do you know what they call a Quarter Pounder with cheese in France? They don't call it . They got the metric system. They call it a "royale with cheese".

[adriangibbons]

2 session data was very suspicion. x "avg_speed": 5.619, Garmin_connect is 20.2 x "max_speed": 13.52, Garmin_connect is 48.7 x "total_ascent": 366, Garmin_connect is 1008

I've just made the setUnits() function generic to session, lap and records messages. So this should be resolved now :)

Hopefully I'll be able to spend some time today developing gearChanges()

[adriangibbons]

Hey, have a look at the output from the gearChanges() function now in the Di2-D-Fly-gearChange branch.

Interested to hear your thoughts on this. Using the combined_summary array, I was able to reasonably quickly create the following visualisation with help from https://gist.github.com/mbostock/4063269

Not sure that we need all of:

1. front_gear_summary[]
2. rear_gear_summary[]
3. combined_summary[]
4. gears_array[]

[takenokotaketake]

Very Good ,Sir. please look at my opnions, this is excel yet. but now drowing by google charts.

[adriangibbons]

Wow - you're running a crazy wide MTB cassette there - are the hills very steep where you are?!

I think the data you need to create such a chart is now available to you - are the numbers for each gear the minutes spent in that combination? e.g. 6min in 33x11t; 25min in 50x11t; etc?

Btw, I think you have 50/34t chainrings, not 50/33t? The file you sent also says this.

I've just closed #22 - I was only checking record messages for invalid_values, not lap, session, event, etc! It would seem there were not valid [event][battery_status] in the file!

[takenokotaketake]

Thank you Sir. it 33t is true, zepher chainrings, I changed, And myhome near the Mt.Aso. very volcanon . I use widecasette, it is very nomaly, but not use teeth in the it. I want search it,but it is not moode, data is true,data was answerd to me. And data is false rand() data,Sir. That is sample. Thanks. And I think battery array is Garmin's work, new, critical, were not bad. garmin was show 0-100 data. They dont have sence. I gonna post garmin foram it. See you Mr.

[takenokotaketake]

Hello , Sir. I am making Di2 data. But array data is into rest time. I dont run F50-R21T alot. because I m caliculate gear summary datta equal $pFFA->data_mesgs['session']['total_timer_time'] Could you Exclusion rest time ? I think somebody man, dosent need gear time on rest time. See you , Mr.

[adriangibbons]

Hey, I can look at this in more detail tomorrow but checkout the function I added isPaused() - just for this purpose! Array indexed by timestamp, true when time was paused, typically due to autopause.

[takenokotaketake]

Very sorry Sir. I dont understand it.

$pFFA->isPaused($pFFA->gearChanges());

[adriangibbons]

It's not that sophisticated unfortunately...

var_dump($pFFA->isPaused());  // doesn't take any parameters

Results in:

array (size=26353)
  799194113 => boolean false
  799194114 => boolean false
  799194115 => boolean false
  799194116 => boolean false
  799194117 => boolean false
  799194118 => boolean false
  799194119 => boolean false
  799194120 => boolean false
  799194121 => boolean false
  ...

There is a boolean for every second starting at the beginning of the ride until the end. true means the time is paused.

So my thinking is that you should be able to ignore the data points where the timer was paused.

NB: the FIT file may not contain data for every second of the ride

For example, you could:

$gear_changes = $pFFA->gearChanges();
$is_paused = $pFFA->isPaused();
foreach ($gear_changes['gears_array'] as $key => $value) {
    if($is_paused[$key] === true) {
        unset($gear_changes['gears_array'][$key]);  // remove data when timer paused
    }
}

[takenokotaketake]

Yes ,Sir. I sucessed. But My opinion is (all user) gears array in not need include paused time. If can exclusion by pause data. but not use your making gear summary. it is very sad. I want to use your gear summary array.

And I changed new php data. it is not use gear_change() I searched txt data, then no function.

[adriangibbons]

Hi @takenokotaketake - I have been very busy this week and not had chance to look at this. What I am thinking of is creating an optional parameter for the gearChanges() function, which will enable the user to request data for all timestamps or only timestamps where the timer was not paused. This will also then be reflected in the summary arrays.

The pictures of what you are trying to achieve are helping me understand what you are trying to achieve.

Thanks for using php-fit-file-analysis :)

[adriangibbons]

Hi @takenokotaketake - I have just updated gearChanges() in the https://github.com/adriangibbons/php-fit-file-analysis/tree/Di2-d-fly-gear-change branch. By default, timestamps where the timer is paused are now excluded. Users could include them by using gearChanges(false) if so desired.

front_gear_summary[]
rear_gear_summary[]
combined_summary[]
gears_array[]

Are all updated accordingly. Hope this helps!?

[takenokotaketake]

Yes ,Sir. I sucessed. Very thank you Sir. gear all time is near the run summary. I understand it gearChanges(true) <= include stop time gearChanges(false) <= run only gearChanges() <= run only

And my hopes idea are many. but you looks like too busy. there are next stage Sir. e.g 1 bike name(garmin edge input data) 2 heat rate zone bpm(1,2,3,4,5) now was max only, I gonna wrote zone line in glaph. 3 gear info

3 is very important It looks like your 'front_gear_summary' and 'rear_gear_summary' were boolean. but we wrote sprocket status. If we run very flat course. inner gear teeth is not wrote we cannot wrote not using sprocket teeth. And I think array number start are dont use [0]. then equall speed number to array number. 'front_gear_info' => array (size=2) [1] => int 50 [2] => int 34 'rear_gear_info' => array (size=11) [1] => int 23 [2] => int 21 [3] => int 19 [4] => int 18 [5] => int 17 [6] => int 16 [7] => int 15 [8] => int 14 [9] => int 13 [10] => int 12 [11] => int 11 They dont hurry Sir. I am enough RGDS

[adriangibbons]

Hi @takenokotaketake

To your three points:

1 bike name(garmin edge input data) I have had a look at the FIT standard profile.xls and I think this should be stored in: Message: bike_profile (mesg_num: 6) Field Name: name (Field Def Num: 0; Field Type: string)

I will take a look at the FIT file you shared with me before, but I don't think that this information is stored in there or I would've added it to the $data_mesg_info array in phpFITFileAnalysis.

Have you seen this information when uploading to a site such as Strava, Garmin Connect, etc?

2 heart rate zone bpm(1,2,3,4,5) now was max only. I have an open issue #27 that I think will take care of this. Which message are you looking at?

Not all Garmin's store this information in the FIT file they create, so phpFITFileAnalysis provides a couple of functions to determine time spent in zones based on HRmax or HRreserve formulae:

$pFFA->hrPartionedHRmaximum($hr_maximum)
$pFFA->hrPartionedHRreserve($hr_resting, $hr_maximum)

These may help you in the interim?

3 gear info I have had a look at the FIT standard profile.xls and I think this should be stored in: Message: bike_profile (mesg_num: 6) Field Name: front_gear_num (Field Def Num: 38) Field Name: front_gear (Field Def Num: 39) [ARRAY] Field Name: rear_gear_num (Field Def Num: 40) Field Name: rear_gear (Field Def Num: 41) [ARRAY]

Again, I am not sure the bike_profile message is in the FIT file you shared with me but I will look again at the weekend. If only gear_change_data event messages are available, then it will not be possible to tell what gears are present on the bike unless they are changed into at some point during a ride.

As to whether the arrays should be numbered from 0 or 1, I am not sure. The FIT profile starts the arrays from 0, i.e. "Number of teeth on each gear 0 is innermost", so I will probably stick with the FIT standard on this one. It's easy enough to +1 when displaying the data to the user. This will avoid some arrays starting at 0 and others starting at 1, which could be confusing to other coders in the future.

'front_gear_summary' and 'rear_gear_summary' should not be boolean, they should be integers. If you have booleans, please could you share the FIT file you are experiencing this with (email to same address as before) and I will look at it.

Thank you, Adrian.

[takenokotaketake]

Hello Sir.

1. No, Sir, I looked bike name in your debag text. I searchd looking my inputed name there. I couldn't read this message because it was full of garbling at debag text. about "private 'file_contents' => " (I dampped and look $pFFA before en

code data)

2. Oh ,Sir, I see. garmin is setting zone persentage area caliculkate by max heartrate.but it is garmin's job. very thanks.
3. gear info is not boolean? No, Sir. look at this. I runned in my house. using 11 speed. but they dont wrote dont use teeth. I want to make gear Analysis sys. more use gear and dont use gear. But this boolean array is not catch all teeth. garmin edge is inputed teeth data. it is not seen there? And I think dont like this array order , array is rand(). I want ASC or DESC. rear(front)_gear_summary, is DESC, good. but combined summary is rand().teeth order foreach $key. I setted graph. then use other arrays order nest combined value array I think summary (front gear, rear gear, combined) is order by $key DESC very beautiful.

But there were problem is not sloving slow, please . Please give priority to your(Sir's) circumstances

[adriangibbons]

file_contents is the literal contents of the file as read into memory. As FIT files are binary, that is why it looks all garbled-up. It would look the same if you opened the FIT file in Notepad. It is a private member as it is not intended for people to access directly for this reason. PHP's unpack() function is used by the class to turn the binary data into something that can be human-read.

You should take a look at:

// contains all the definition messages found in the file
var_dump($pFFA->defn_mesgs_all);

// contains all the data from the file (that has a definition message understood by the class)
var_dump($pFFA->data_mesgs);

The $pFFA->showDebugInfo() function outputs a useful table with debug information containing the above.

Hope this helps!

[takenokotaketake]

Hello, Sir. Could you pick up road grade(percent) array ? every second or timestamp.

[takenokotaketake]

Hello, Sir. I slove to 11 Sep at today.

[adriangibbons]

Hi @takenokotaketake

Could you pick up road grade(percent) array ?

The class already listens for grade: it's field definition 9 in record messages:-

https://github.com/adriangibbons/php-fit-file-analysis/blob/master/src/phpFITFileAnalysis.php#L779

20 => [
    'mesg_name' => 'record', 'field_defns' => [
        0 => ['field_name' => 'position_lat',          'scale' => 1,    'offset' => 0,        'units' => 'semicircles'],
        1 => ['field_name' => 'position_long',         'scale' => 1,    'offset' => 0,        'units' => 'semicircles'],
        2 => ['field_name' => 'altitude',              'scale' => 5,    'offset' => 500,    'units' => 'm'],
        3 => ['field_name' => 'heart_rate',            'scale' => 1,    'offset' => 0,        'units' => 'bpm'],
        4 => ['field_name' => 'cadence',               'scale' => 1,    'offset' => 0,        'units' => 'rpm'],
        5 => ['field_name' => 'distance',              'scale' => 100,  'offset' => 0,        'units' => 'm'],
        6 => ['field_name' => 'speed',                 'scale' => 1000, 'offset' => 0,        'units' => 'm/s'],
        7 => ['field_name' => 'power',                 'scale' => 1,    'offset' => 0,        'units' => 'watts'],
        9 => ['field_name' => 'grade',                 'scale' => 100,  'offset' => 0,        'units' => 'percent'],
        10 => ['field_name' => 'resistance',           'scale' => 1,    'offset' => 0,        'units' => ''],
        13 => ['field_name' => 'temperature',          'scale' => 1,    'offset' => 0,        'units' => 'C'],
        39 => ['field_name' => 'vertical_oscillation', 'scale' => 10,   'offset' => 0,        'units' => 'mm'],
        40 => ['field_name' => 'stance_time_percent',  'scale' => 100,  'offset' => 0,        'units' => 'percent'],
        53 => ['field_name' => 'fractional_cadence',   'scale' => 128,  'offset' => 0,        'units' => 'rpm'],
        253 => ['field_name' => 'timestamp',           'scale' => 1,    'offset' => 0,        'units' => 's']
    ]
],

If grade messages are in the FIT file then they will be available through:

var_dump($pFFA->data_mesgs['record']['grade']);

You may see gradient in websites such as Strava and Garmin Connect, even though the gradient information was not in the FIT file. I think they are calculating the grade using changes in distance and altitude.

[adriangibbons]

every second or timestamp.

Are you saying that you are interested in issue #32 ?

[adriangibbons]

I slove to 11 Sep at today.

I'm not sure what you mean. Yes it's been while since my last message on the 11 Sep, but I have limited free time and have prioritised other things.

[takenokotaketake]

No, Thank you. it is cheked and slove.

[adriangibbons]

I have implemented a data_every_second option and closed #32. Note that if there are a lot of missing timestamps then performance can be degraded.

[takenokotaketake]

Hello,Sir. gearChanges() is deleted new version?

[adriangibbons]

Hi - it hasn't been deleted; I've just never merged gearChanges() into the master branch. It has only existed in the Di2-d-fly-gear-change branch, which is now quite a few commits behind the master.

If gearChanges() is working for you(?), then perhaps I just need to tidy a few things up and merge it into the master.

[adriangibbons]

I've added gearChanges() to the master branch and removed the Di2-d-fly-gear-change branch.

Could I include the FIT file you sent me online in the Github project? I could do with adding a PHP Unit test for the gearChanges() function.

Cheers, Adrian.