New activity model with different sleep classification

[jwang-lilly]

It appears sleep classification has changed significantly between the new models and previous models. I'd greatly appreciate if you could explain what are some of the key improvements. Thanks much.

new models: doherty-may20.tar willetts-may20.tar

old models doherty2018.tar willetts2018.tar

[chanshing]

Hi, can you describe what differences you see in the results? The new models were updated to reflect some changes in the preprocessing/feature engineering. Also, are you using sampleRate 100?

[jwang-lilly]

I'm using sampleRate 50 consistently. 1) Very different in what is considered as sleep vs sendentary. In some cases, assignment appears to be reversed; 2) sleep seems to requires more extended/continous inactivity in the new algorithm vs old one which has very fragmented sleep call. Just want mention that acceleration stays the same so preprocessing/feature engineering doesn't seem to explain the major difference.

[chanshing]

The acceleration will stay the same, but the extracted features may change. You can inspect these by passing --deleteIntermediateFiles False which will keep the intermediate file *-epoch.csv from being deleted. This file contains the extracted features (such as signal mean, variances, FFT coefficients and others) that the internal machine learning model uses to make the predictions. The machine learning model is a random forest trained on these features. There was a recent change in how the features are computed, so a new random forest model had to be trained (doherty-may20). I suspect that what you are seeing is an issue with the sampling rate, because some features are not sample-rate-independent. Please can you try sampleRate 100 with the new model? Also, can you try the old model, also with sampleRate 100, but with an earlier commit of the current code? (at least ef7e651b67235d76d1065fdb39d9ba8b91a7326d )

[jwang-lilly]

Thanks much for the prompt feedback. I did old, new, 50 Hz and 100 Hz 4-way comparison. As you stated, there is a difference between 50 Hz and 100 Hz sampling rate but it is tiny and barely discernable. But the difference between old (doherty2018-apr20Update) and new (doherty-may20) are drastic. Large sections of sedentary are turned into sleep in the new model. I have to admit that sleep makes more sense as a result but seeing so much has been changed is concerning without knowing what is going on.

[chanshing]

Thank you for the comparisons. Please can you confirm that you run the old model using version https://github.com/activityMonitoring/biobankAccelerometerAnalysis/commit/ef7e651b67235d76d1065fdb39d9ba8b91a7326d ? I suspect that you might need to go back earlier, perhaps 55c8ac91ea57385bf4327c4ef5f9e804a08ac0b4 Please would you mind trying the old model again at this commit?

[jwang-lilly]

Thanks chanshing. The models themselves are not in github. There are three sets of models 1) wget ${downloadDir}doherty2018.tar -P activityModels/ wget ${downloadDir}willetts2018.tar -P activityModels/ 2) wget ${downloadDir}doherty2018-apr20Update.tar -P activityModels/ wget ${downloadDir}willetts2018-apr20Update.tar -P activityModels/ 3) wget ${downloadDir}doherty-may20.tar -P activityModels/ wget ${downloadDir}willetts-may20.tar -P activityModels/

1) and 3) have similar results but different from 2)

[chanshing]

I see... so doherty2018.tar should run with 55c8ac91ea57385bf4327c4ef5f9e804a08ac0b4 doherty2018-apr20Update should run with c989ad80a4f3fc039330f94af51a291ddc07e84a doherty-may20.tar should run with latest commit

I think there were issues with the first and second with respect to sensibility to sampling rate but I am not sure. You mentioned that you didn't see much difference between sampleRate 50 and 100

[aidendoherty]

Hi @JianWang2016 and @chanshing - many thanks for this very helpful discussion.

@JianWang2016 - as @chanshing has mentioned, we have made updates to how some of the features are computed. These fix some obvious minor bugs, but do result in improved classification (kappa of 0.77 for new vs 0.74 for old model on our CAPTURE-24 dataset).

The other change we have made is to reduce the number of trees in the random forest (from 1000 to 100 trees). Classification performance remains the same, but the size of the model is much smaller. We have therefore decided to go with the smaller 100 trees model.

Can you confirm that the new classifications appear to have better face validity? i.e. they intuitively look more correct?

[jwang-lilly]

Thanks much to @chanshing and @aidendoherty

I can confirm the new classifications appear to have more sensible sleep classification. I am closing this issue now.

[aidendoherty]

Many thanks for this helpful feedback @JianWang2016