Activity calories in the database are high.

[wmbaum]

Almost all of my activity calories in the database are significantly higher than in the fit files.

One particular fit file, 4635447473.fit, shows 498 calories in Garmin connect, and also shows "calories": 498.0 in activity_4635447473.json and summaryDTO.calories: 498.0 in activity_details_4635447473.json, yet the garmin_activities.acivities.calories shows 798.213134765625. The summary records also reflect the higher values for activities_calories.

I've seen a couple of activities where the calories match, but the vast majority are significantly higher.

Is there some adjustment being done to activity calories that I'm not aware of, or should they match the value in the fit file?

[tcgoetz]

I just went through a sample of the activities in my database and garmin_activities.activities.calories matches activitydetails*.json. So we need to dig into why that is not the case for you.

• Is it wrong for all of your activities or just some? If some we need to dig into the pattern of which activities are wrong. If all, then we need to dig into a wider pattern. For, instance, is your DB set for metric?
• What version of Garmin DB?

Besides the activitydetails*.json files, the calories fields is also populated from .fit files. FIT files also for developer extensions. In garmindb develop fields will override base fields of the same name. Any chance that your FIT files for these activities are from a develop app or a gamin app using developer fields?

[wmbaum]

I think you nailed it with the developer fields.. The fit files in question are from custom apps that have a bunch of extra Connect IQ fields. The vast majority of these are from fbbbrown's Strength Training+ app ( https://apps.garmin.com/en-US/apps/2b2e6e1c-81d2-48b5-8f1b-9ca7c77c3b96 ). I have a few other apps of his which seem to be exhibiting the same behavior.

The activity above in Garmin Connect shows 498 "Calories" but 798.2 Cal for "Instantaneous Calorie Sum (hr)", and then 802.5 Cal for "Adjusted Calories (avghr)". I converted the fit file to csv, and I see: 't_cal,"798.21313",cal,t_calAV,"802.49963",cal'.

Would you like me to send you some of these fit files? Email? Connect on Garmin Connect?

[tcgoetz]

Are you saying you don't want it to use the dev field for calories? What is the outcome your looking for?

If you put your FIT file into GarminDb\test\test_files\fit\activity, then cd GarminDB\test, and run make fit_file after which fit_file.log will contains the parsed contents for all of the test FIT files. Search for yours and you can view the contents.

[wmbaum]

No, I do not want to use the dev field for calories. It isn't intended to replace the standard Garmin calorie count, it's just an additional data point of a different calorie measurement algorithm. Garmin Connect, Strava, and all other tools I've tried show the standard calorie count.

I ran the fit file referenced above through the fit_file test. In fit_file.log (which is huge :) ) I see:

`DeveloperFieldDefinition(dev_eE[eE]: scale None offset None, 1 of UINT16) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_RRhr[RRhr]: scale None offset None, 1 of FLOAT32) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_RRin[RRin]: scale None offset None, 1 of FLOAT32) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_HRV[HRV]: scale None offset None, 1 of FLOAT32) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_loc[loc]: scale None offset None, 1 of FLOAT32) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_rMSSD[rMSSD]: scale None offset None, 1 of FLOAT32) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_SDNN[SDNN]: scale None offset None, 1 of FLOAT32) for <MessageType.record: 20> field None DeveloperFieldDefinition(dev_Min_hr[Min_hr]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_Bat[Bat]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_Stps[Stps]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_total_calories[t_cal]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field 11 DeveloperFieldDefinition(dev_t_calAV[t_calAV]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_tSets[tSets]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_dBts[dBts]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_sBts[sBts]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_aHRV[aHRV]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_arMSSD[arMSSD]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_aSDNN[aSDNN]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_HRV+/-[HRV+/-]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_tStps[tStps]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_ttime[ttime]: scale None offset None, 8 of CHAR) for <MessageType.session: 18> field None DeveloperFieldDefinition(dev_tcal[tcal]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None

And in 'Message:' total_calories(498.0 kcal), . . . dev_total_calories(798.213134765625 cal), dev_t_calAV(802.4996337890625 cal)

When I view the csv (from the FIT SDK from thisisant.com) I just see' t_cal,"798.21313",cal' and 't_calAV,"802.49963",cal' and the definitions are:

Data,14,field_description,field_name,"t_cal",,units,"cal",,native_mesg_num,"18",,developer_data_index,"0",,field_definition_number,"6",,fit_base_type_id,"136",,native_field_num,"11",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, Data,14,field_description,field_name,"t_calAV",,units,"cal",,native_mesg_num,"18",,developer_data_index,"0",,field_definition_number,"7",,fit_base_type_id,"136",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

We can see above that "t_calAV" is being interpreted as dev_t_calAV[t_calAV], but for some reason "t_cal" is being translated into dev_total_calories[t_cal] instead of just (dev_t_cal[t_cal].

Would you like to see the original fit file or log?

[tcgoetz]

I disagree. The developer explicitly defined that field to shadow the native total_calories field.

DeveloperFieldDefinition(dev_total_calories[t_cal]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field 11

Comes from: https://github.com/tcgoetz/Fit/blob/26fa50709c02da87e9bb4f61558f852b467d741c/developer_field_definition.py#L69

And what that is saying is that the developer defined that field to shadow native field: <MessageType.session: 18> field 11

which is total_calories.

[tcgoetz]

It's also worth noting that, besides defining a total calories field that shadowed the native field, they also defined a field with the same parameters as not shadowing:

DeveloperFieldDefinition(dev_tcal[tcal]: scale None offset None, 1 of FLOAT32) for <MessageType.session: 18> field None

[wmbaum]

Interesting. I guess the distinction is between shadowing and overriding.. This is different behavior, as Garmin Connect clearly does not see it as overriding:

498 C Calories ... 798.2 Cal Instantaneous Calorie Sum (hr) 802.5 Cal Adjusted Calories (avghr)

I suppose some may want this behavior, but I would think most would want GarminDB to produce numbers that match what's in Garmin Connect, MyFitnessPal, etc..

Thanks for showing me where that is. This will make it easy for me to disable. For my purposes, having the total calories overridden is a deal breaker -- and I have little to no interest in any of the data in these dev fields. I'd be fine with not importing them at all -- or even skipping the fit files and just using the data from the activities json files. But.. now that I know where the switch is, I can flip it off.

[tcgoetz]

I can add a config item to ignore developer fields, but it will not be the default.

Personally I use developer apps and data fields to get the extra data they provide. I'm not sure why you would use them if you didn't want the data.

[wmbaum]

I use these apps because I like the display and the functionality during the activity. For the Strength+ app, I particularly like the rest timer, which I use between every set.

Also, it's not that I don't want the extra data from the dev fields, but I cannot have critical data destructively overwritten by a decision to rename a dev field. Extra data is great -- overwriting critical data is not.

Sorry for the delay in the reply.. The option to ignore the dev fields is great, but I think safer and simpler might be an (additional?) option to never rename a dev field over the top of an existing field? Personally I think the default should allow the data to come through in a way that is compatible with how the data is interpreted by Garmin Connect.

[tcgoetz]

The Fit decoding doesn't overwrite anything. The fact that the dev field that shadows totalcalories is prepended with dev and becomes dev_total_calories avoids that issue. The message dictionary for a session message in the above case will contain both total_calories and dev_total_calories. The change in 3f25e69 overrides the code in the data base load that prefers fields named dev_x to x.

[tcgoetz]

Please try the above change and let me know if it works for you.

[wmbaum]

Hmm.. still looks the same. Pretty sure I got the config right, etc., but I'm still seeing the larger calorie numbers. I'll try it again with a small set of fit files. Doing all of them takes many hours...

[wmbaum]

Got it working.. There is a conflict between garmin_connect_config_manager.py which is looking for a config section named 'modes' and GarminConnectConfig.json.example, which had it as "config". Switch "config" to "modes" and it works fine.

I also tested whether or not skipping the field rename in Fit\developer_field_definition.py would produce the desired results and that works as well.

[tcgoetz]

There is a conflict between garmin_connect_config_manager.py which is looking for a config section named 'modes' and GarminConnectConfig.json.example, which had it as "config". Switch "config" to "modes" and it works fine.

Fixed