search

ojwalch / sleep_classifiers

Classify sleep from heart rate and acceleration via Apple Watch
183 stars 84 forks source link

the random seed for parameter selection and step data #23

Open stylebased opened 2 years ago

stylebased commented 2 years ago

Hi, I am really impressed by your paper. When I tried to reproduce your results with your codes and dataset, the results looks different from your results in paper eg. Figure4. Can I assume this difference is caused by randomness in hyperparameter-selection? If so, what is the random seed you used that I can exactly reproduce your results? Best

ojwalch commented 2 years ago

Hi! Tragically, I didn't keep track of the random seed when I was doing this work—my bad! How different are the plots looking? Feel free to reach out at ojwalch @ umich.edu if you'd like to follow-up there.

Best, Olivia

stylebased commented 2 years ago

Thank you for your quick reply.This is the figure 3 in your paperimage. This is the plot I got with same title

image

. Does this difference make sense? Did you use Monte Carlo hyper-parameter searching method?

ojwalch commented 2 years ago

Yep, that difference makes sense: For those figures, I'm randomly choosing the training and testing set, evaluating performance, and then repeating the process many times. The random choices of training/testing set will cause variation in this figure, as will the number of times you repeat the simulation (I believe I did 200 times for the paper). Hope this helps!

stylebased commented 2 years ago

Your response is really helpful. I notice there are two hyper-parameter selection method in your codes. One is Leave-One-Out and another is Monte Carlo. Which one did you use in your paper figures such as figure 2, figure 3. Thanks

ojwalch commented 2 years ago

Both Figures 2 and 3 are Monte Carlo cross validation 👍

stylebased commented 2 years ago

thanks

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Olivia Walch @.> 发送时间: Thursday, July 7, 2022 9:13:54 PM 收件人: ojwalch/sleep_classifiers @.> 抄送: Hu, Xiangyu @.>; Author @.> 主题: Re: [ojwalch/sleep_classifiers] the random seed for parameter selection and step data (Issue #23)

Both Figures 2 and 3 are Monte Carlo cross validation 👍

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fojwalch%2Fsleep_classifiers%2Fissues%2F23%23issuecomment-1178427332&data=05%7C01%7Cxiangyu%40email.sc.edu%7C3e19c4c3aef3467ac09a08da607f20c5%7C4b2a4b19d135420e8bb2b1cd238998cc%7C0%7C0%7C637928396380098413%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ux%2FGEuyQs7Ud52iL1PBvueOyF4K4ZOdqFjxed687mw0%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK7FNGB6GQ6OXMNXSNXQP73VS56FFANCNFSM526LM2YA&data=05%7C01%7Cxiangyu%40email.sc.edu%7C3e19c4c3aef3467ac09a08da607f20c5%7C4b2a4b19d135420e8bb2b1cd238998cc%7C0%7C0%7C637928396380098413%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cw2A8%2F3OSzNc0f3XBsf0wbSSmuOIKxw2tRrnAOO1kXc%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

stylebased commented 2 years ago

I have another question. In your 'clock proxy' and time-based feature part, you used the steps to estimate the light. Can you provide me more detail things about how did you use steps data to predict the circadian clock? For example, what is the threshold value you set to say there is light? In your paper, you said 'if steps were above a threshold, the light was assumed to be one of three levels'