Fitbit_Worn is missing for some participants in daily-metrics.csv

[mengeks]

When we want to use Fitbit_Worn in daily-metrics.csv to calculate the start date for all participants, we have encountered missingness in Fitbit_Worn. Fitbit_Worn is completely missing for some participants, eg: 10032

@samurphy11 has also pointed out that in the HeartSteps 2.0 Application Architecture document, under daily-metrics.csv, it says "Fitbit Worn....If blank HeartSteps is missing data for that participant on that day". It also says "Fitbit Step Count...If blank HeartSteps is missing data for that participant on that day, which should be considered a bug."

This two statements seem ambiguous --- what does it mean by "HeartSteps is missing data"? We observe that for id=10032, Fitbit_Worn is completely missing, but Fitbit Step Count is full of values.

[samurphy11]

@Nick Reid nickreid@nickreid.com @Pedja Klasnja klasnja@umich.edu it would be great if we could get this confusion straight as we are unclear right now when people start being randomized (we thought it was after the user has 7 days in which fitbit is worn 8(?) hours) but the data in the csv files is not consistent with this interpretation. Thanks! Susan

On Mon, Nov 9, 2020 at 3:51 AM mengeks notifications@github.com wrote:

When we want to use Fitbit_Worn in daily-metrics.csv to calculate the start date for all participants, we have encountered missingness in Fitbit_Worn. Fitbit_Worn is completely missing for some participants, eg: 10032

@samurphy11 https://github.com/samurphy11 has also pointed out that in the HeartSteps 2.0 Application Architecture document, under daily-metrics.csv, it says "Fitbit Worn....If blank HeartSteps is missing data for that participant on that day". It also says "Fitbit Step Count...If blank HeartSteps is missing data for that participant on that day, which should be considered a bug." [image: Screen Shot 2020-11-09 at 12 44 39 AM] https://user-images.githubusercontent.com/12406669/98518938-f20c6e00-2224-11eb-80e5-bb9d96c4eeba.png

This two statements seem ambiguous --- what does it mean by "HeartSteps is missing data"? We observe that for id=10032, Fitbit_Worn is completely missing, but Fitbit Step Count is full of values.

[image: Screen Shot 2020-11-09 at 12 41 25 AM] https://user-images.githubusercontent.com/12406669/98518600-70b4db80-2224-11eb-9346-8f667504d585.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kpwhri/heartsteps/issues/194, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTRRQJWL3LPMHP4FOP6B2TSO6UQNANCNFSM4TPBJ4JQ .

--

---

Susan A. Murphy Professor of Statistics, Radcliffe Alumnae Professor at the Radcliffe Institute and Professor of Computer Science at the Harvard John A. Paulson School of Engineering and Applied Sciences Harvard University website: http://people.seas.harvard.edu/~samurphy/ http://people.seas.harvard.edu/~samurphy/

---

[nickdotreid]

I have just pushed an update that will fix this issue. The root of the problem, is that this participant started (and ended apparently) before the daily adherence messages application was added to HeartSteps -- which caused columns in the daily_metrics export that used data generated by daily adherence metrics to be empty. These columns were "Fitbit Worn" "Fitbit Updated" and "App Used"

Rather than generate daily adherence metric data, I decided to query and compute these columns directly.

One change I made, was to report "Fitbit Update Count" rather than "Fitbit Updated" -- The Fitbit Update Count is the number of times the Fitbit API pushed an updated to the HeartSteps server on a specific day.

I will be updating the Application Architecture document shortly.

[samurphy11]

Thanks so much @Nick Reid nickreid@nickreid.com !! Susan

On Wed, Nov 18, 2020 at 3:38 PM Nick Reid notifications@github.com wrote:

I have just pushed an update that will fix this issue. The root of the problem, is that this participant started (and ended apparently) before the daily adherence messages application was added to HeartSteps -- which caused columns in the daily_metrics export that used data generated by daily adherence metrics to be empty. These columns were "Fitbit Worn" "Fitbit Updated" and "App Used"

Rather than generate daily adherence metric data, I decided to query and compute these columns directly.

One change I made, was to report "Fitbit Update Count" rather than "Fitbit Updated" -- The Fitbit Update Count is the number of times the Fitbit API pushed an updated to the HeartSteps server on a specific day.

I will be updating the Application Architecture document shortly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kpwhri/heartsteps/issues/194#issuecomment-729939772, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTRRQK5VZGP7B673UYTMULSQQWFDANCNFSM4TPBJ4JQ .

--

---

Susan A. Murphy Professor of Statistics, Radcliffe Alumnae Professor at the Radcliffe Institute and Professor of Computer Science at the Harvard John A. Paulson School of Engineering and Applied Sciences Harvard University website: http://people.seas.harvard.edu/~samurphy/ http://people.seas.harvard.edu/~samurphy/

---

[mengeks]

Thanks @nickdotreid !