[review] Uncertainty estimates

[ppxasjsm]

This one is for @JenkeScheen and @hannahbrucemacdonald

In 7.2.5, concerning uncertainty estimates. I found it helpful for many students to realize that free energy error estimates often suffer from exactly the same sampling problems as the free energy estimate itself. Specifically, if the free energy estimate is off because an important region in configuration space has not been sampled, the error estimate has no way to indicate this. The authors might want to make this point super-clear.

[agrossfield]

As a non-author, I think this point is incredibly important. More to the point, internal uncertainty estimates are almost guaranteed to underestimate the statistical error (relative to repeated runs with distinct starting structures, the gold standard), because they’re based on the assumption that you’ve seen everything.

---

Dr. Alan Grossfield Dept of Biochemistry and Biophysics University of Rochester Medical Center Phone: 585 276 4193 http://membrane.urmc.rochester.edu https://orcid.org/0000-0002-5877-2789 Pronouns: He/his

From: Toni Mey notifications@github.com Reply-To: alchemistry/alchemical-best-practices reply@reply.github.com Date: Monday, September 28, 2020 at 7:31 AM To: alchemistry/alchemical-best-practices alchemical-best-practices@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [EXT] [alchemistry/alchemical-best-practices] [review] Uncertainty estimates (#105)

This one is for @JenkeScheenhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JenkeScheen&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=Lg2GEvSNZVygV5eq7KnI46MWPqu7oGYmJvU5hcVTC1Y&s=7QpRBx2OPRvY03pMbLB4GuCkujCaRGc1v3jOqp6DAPw&e= and @hannahbrucemacdonaldhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hannahbrucemacdonald&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=Lg2GEvSNZVygV5eq7KnI46MWPqu7oGYmJvU5hcVTC1Y&s=AZdf9UmliEv4ouKIDazNKw-FnxXR6OGsywXMLm5p4qI&e=

In 7.2.5, concerning uncertainty estimates. I found it helpful for many students to realize that free energy error estimates often suffer from exactly the same sampling problems as the free energy estimate itself. Specifically, if the free energy estimate is off because an important region in configuration space has not been sampled, the error estimate has no way to indicate this. The authors might want to make this point super-clear.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_alchemistry_alchemical-2Dbest-2Dpractices_issues_105&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=Lg2GEvSNZVygV5eq7KnI46MWPqu7oGYmJvU5hcVTC1Y&s=TG2KI8q62MzP-EFlIxadJUE8aS7MTCqZSmojEE-ZUPk&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADH754RPNCECFMZPVCTHQD3SIBXZ5ANCNFSM4R4MPJQA&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=Lg2GEvSNZVygV5eq7KnI46MWPqu7oGYmJvU5hcVTC1Y&s=32NjC-Wc_bFFYUSQ5u5DTqUoA1J6-vU-2jPXJUGD40Q&e=.

[mrshirts]

I'm happy to add something on this as well (I will defer to other people to address this first since I have a number of other tasks).

[davidlmobley]

Yes, it's important to state very clearly that if you've never seen something, you have NO WAY AT ALL of knowing you have never seen it. (I use a hiking analogy; until you climb to the top of some ridge you have no idea at all about what is in the next valley nor whether it even exists. No amount of information you collect about your initial valley can tell you how many other valleys are adjacent.)

[hannahbrucemacdonald]

I've added two sentences, but it felt better putting in Section 8.4 rather than 7.2.5

[hannahbrucemacdonald]

Also added something in 7.2.5

[ppxasjsm]

Thank you!