search

dmzuckerman / Sampling-Uncertainty

Best Practices article intended for LiveCoMS
36 stars 5 forks source link

Use of "random" in the glossary #24

Closed agrossfield closed 6 years ago

agrossfield commented 6 years ago

The word "random" is used a bunch of times in the definitions of key terms in a way that makes no sense to me. For example, "independent observables" are defined as "Random quantities that are uncorrelated". These quantities are presumably calculated from coordinates in a deterministic way, so referring to them as random is confusing.

I understand that when we do statistics on stuff we're treating the snapshots as random samples, but this nomenclature is not going to be helpful when dealing with new scientists eager to learn.

agrossfield commented 6 years ago

The more I think about it, the more worried I am about the whole section. It's formally correct, but it's not very readable. I think the remarks need to be more extensive, and translate the definitions into more familiar wordings. Otherwise, I'm really worried that much of our intended audience (or at least the biomolecular simulation folks, which is the group I'm most familiar with) will just stop reading there.

dwsideriusNIST commented 6 years ago

Alan, we're definitely not in a final state for that section. Currently it represents a very mild rewriting of the exact definitions in the VIM, so it's still a lot pedantic. @mangiapasta and I have been unable to work on it for the past few weeks. We're planning to get to it after another week of down time.

agrossfield commented 6 years ago

Totally cool -- I figured that might be the case. I was just worried that the biomolecular community, which is in my experience less comfortable with statistical rigor and terminology than the materials folks, might get lost or turned off, and that since you're coming from a more rigorous place to start you might not realize.

If you want me to take a crack at it, I'm willing to do some editing, but I didn't want to step on toes by editing in place.

I apologize if I insulted or offended you -- that was not my intent.

Cheers,

Alan

dwsideriusNIST commented 6 years ago

No problem, no offensive even considered on my part. A major part of our goal for that section is to introduce more statistical rigor into simulations in general, so we're trying to find a balance between the pedantic VIM definitions and more practical terminology. I think your anticipation that the biophys community is not as far along as materials/fluids, but don't give materials/fluids too much credit. There's a lot of really bad uncertainty analysis out there. Most people don't do error analysis and those that do it correctly are a pretty small fraction.

Please hold off on wordsmith-style editing for a bit so that we can expand it. If you have some specific thoughts, please do add remarks or comments in the text for us! (I typically do fetch/pull before resuming work.)

agrossfield commented 6 years ago

Sounds good, Dan. I always worry critiquing via email because there's no facial expressions or tone of voice to keep it light.

If I have specific ideas, I'll add them as you say, most likely in the remarks sections.

Dr. Alan Grossfield Department of Biochemistry and Biophysics University of Rochester Medical Center

On Nov 10, 2017, at 7:57 PM, Daniel W. Siderius notifications@github.com<mailto:notifications@github.com> wrote:

No problem, no offensive even considered on my part. A major part of our goal for that section is to introduce more statistical rigor into simulations in general, so we're trying to find a balance between the pedantic VIM definitions and more practical terminology. I think your anticipation that the biophys community is not as far along as materials/fluids, but don't give materials/fluids too much credit. There's a lot of really bad uncertainty analysis out there. Most people don't do error analysis and those that do it correctly are a pretty small fraction.

Please hold off on wordsmith-style editing for a bit so that we can expand it. If you have some specific thoughts, please do add remarks or comments in the text for us! (I typically do fetch/pull before resuming work.)

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmzuckerman_Sampling-2DUncertainty_issues_24-23issuecomment-2D343625906&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=IghF34pe82mYwsPt4OTr5zRnZaTUd90NTtvmccR1BsM&s=KIsPI5KAwP7L0b3XB0eHMb9mnrRR36YGWdg023nxFzg&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AM-5F-2D8kLrBrO5-2DPOoq2y0MOqCOMcuSNp0ks5s1PEKgaJpZM4QaHwR&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=IghF34pe82mYwsPt4OTr5zRnZaTUd90NTtvmccR1BsM&s=7IXUoShkU66Gt1cb8bh_iQBL2oVjni8QTNGDpBlPFqE&e=.

dmzuckerman commented 6 years ago

Thanks a lot @agrossfield for the useful comments and @dwsideriusNIST and @mangiapasta for working on this section - which I wrongly thought would be straightforward!

I agree with @agrossfield that we'll need to be careful that our plain-English remarks are plain enough. I'll be happy to review those once @dwsideriusNIST and @mangiapasta give the ok.

A few specific suggestions:

mangiapasta commented 6 years ago

Apologies for radio silence the past few weeks. I've been swamped with some internal deadlines.

To follow up on some of these issues:

1) @agrossfield and others. The confusion you highlight concerning the use of the word "random" is justified. As a somewhat technical aside (which maybe isn't really an aside?), much of the MD enterprise is based on the assumption that Newton's equations are a deterministic random number generator. If you stop to think about that for just one moment, you realize immediately that this is an oxymoron. Moreover, it's not generically true, as the two-body problem demonstrates. So, I think we're banking on the fact that the chaotic behavior of many-body systems looks a lot like randomness. Proving this mathematically is, as far as I know, an open problem.

So, in essence, I think we're using the word "random quantity" as a synonym for "a quantity whose value can't be predicted in practice with complete certainty." The lack of certainty here comes from the strong dependence of the dynamics on the initial conditions, which we can never nail down to better than machine precision. (Also, floating-point addition is not associative. These facts do have very real-world implications for MD simulations in that they break time-reversibility.)

Anyways, one approach is to adopt the definition of random quantity as one whose value can't be predicted with complete certainty and then add a remark about how in MD, randomness is really a practical issue related to chaos. Unfortunately this is something of a cliffhanger because it opens a rabbit-hole of a question about what we mean by probability in statistical mechanics. But I don't know of a better way to handle this.

2) @dwsideriusNIST I also dislike the usage of "experimental" in terms such as "experimental standard uncertainty" etc. Since we're not VIM, I would be okay with either changing the word to "empirical", putting experimental in parentheses, or removing it altogether and remarking that VIM differs from us in this regard. Thoughts?

3) @dmzuckerman I changed "Independent observables" to "uncorrelated observables." That should make Eq. 4 be correct now. Given how complicated the definitions section is getting, I think it's better to avoid a discussion of independence. From a practical standpoint, it's essentially impossible to demonstrate with simulated data. To within a given threshold, however, we can always compute correlations.

dmzuckerman commented 6 years ago

@mangiapasta a couple of comments:

FYI most biomolecular MD is done with a thermostat. As you may know, there are both deterministic and (pseudo-)random thermostats out there; the most popular are actually deterministic (which I don't like because I think the thermal 'bath' is unknowable and effectively infinite, so physically that seems the essence of random to me).

Regarding Eq (4) I'm not sure there's a difference between the technical meanings of 'independent' and 'uncorrelated' (though I could be wrong). I think you're thinking of linear correlation ... which Eq (4) indeed addresses correctly. But I don't think we want to perpetuate the common misconception that linear correlation is the only kind one needs to worry about.

Here's a simple example of a distribution with no linear correlation: p(x,y) ~ exp[ -( r - r_0 )^2 ], where r = sqrt( x^2 + y^2 ), which is a ring-shaped distribution. The variables x and y are correlated because the x value affects the y distribution and vice versa, even though < x y > = = 0.

Given the complexity of systems our communities are dealing with, I don't think we want to restrict ourselves to linear correlations.

That being said, I agree we don't want to get into a long discussion of independence. We need to refer readers out to other discussions. But I do think we need to give the straight-up defn of independence, which is p(x,y) ~ p(x) * p(y)

agrossfield commented 6 years ago

Minor comment: these days, the vast majority of biomolecular simulations are done with pseudo-random thermostats. GROMACS has Nose-Hoover, but NAMD and Amber don’t, and rely on Langevin thermostatting. Not sure about OpenMM, but that’s a smaller user community at this point. Even in GROMACS, many people use Langevin, though an appalling number still use Berendsen.

Aaln

On Nov 16, 2017, at 10:28 AM, dmzuckerman notifications@github.com<mailto:notifications@github.com> wrote:

@mangiapastahttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mangiapasta&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=ar0Ck8UYelvgkfWPrRoRgKgQLr8JyPll8TlR0nAjP7s&s=7MgQbOL9foOrwazN7Q111SCsqBw2igNSqOwq0y5_JnA&e= a couple of comments:

FYI most biomolecular MD is done with a thermostat. As you may know, there are both deterministic and (pseudo-)random thermostats out there; the most popular are actually deterministic (which I don't like because I think the thermal 'bath' is unknowable and effectively infinite, so physically that seems the essence of random to me).

Regarding Eq (4) I'm not sure there's a difference between the technical meanings of 'independent' and 'uncorrelated' (though I could be wrong). I think you're thinking of linear correlation ... which Eq (4) indeed addresses correctly. But I don't think we want to perpetuate the common misconception that linear correlation is the only kind one needs to worry about.

Here's a simple example of a distribution with no linear correlation: p(x,y) ~ exp[ -( r - r_0 )^2 ], where r = sqrt( x^2 + y^2 ), which is a ring-shaped distribution. The variables x and y are correlated because the x value affects the y distribution and vice versa, even though < x y > = = 0.

Given the complexity of systems our communities are dealing with, I don't think we want to restrict ourselves to linear correlations.

That being said, I agree we don't want to get into a long discussion of independence. We need to refer readers out to other discussions. But I do think we need to give the straight-up defn of independence, which is p(x,y) ~ p(x) * p(y)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmzuckerman_Sampling-2DUncertainty_issues_24-23issuecomment-2D344957523&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=ar0Ck8UYelvgkfWPrRoRgKgQLr8JyPll8TlR0nAjP7s&s=okhKCQYSuphGL_55AVtPJipJ21vOlPo0yjRakHlW_KI&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AM-5F-2D8ndb-2DCUBWDIg7MnUzl-2D7gxR-2DYtGhks5s3FSLgaJpZM4QaHwR&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=ar0Ck8UYelvgkfWPrRoRgKgQLr8JyPll8TlR0nAjP7s&s=3g3GZtnQxfFJ9A-VHhYA9cDO0yc_KlKpULogNpaR1OI&e=.

Dr. Alan Grossfield Associate Professor Department of Biochemistry and Biophysics University of Rochester Medical Center 610 Elmwood Ave, Box 712 Rochester, NY 14642 Phone: 585 276 4193 http://membrane.urmc.rochester.edu

dmzuckerman commented 6 years ago

@agrossfield Nose-Hoover is deterministic

agrossfield commented 6 years ago

Right, but it’s almost never used — not implemented in Amber or NAMD, and rarely chosen in GROMACS.

On Nov 16, 2017, at 10:32 AM, dmzuckerman notifications@github.com<mailto:notifications@github.com> wrote:

@agrossfieldhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_agrossfield&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=AMzd0ifqo1knKX-yHVb2L4zhGXYVmXO-yUSuGbO9d2Q&s=bErJFbjjzGyjdwjAYMrNOMPzFAVLkcXzz9eBIHa-gK4&e= Nose-Hoover is deterministic

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmzuckerman_Sampling-2DUncertainty_issues_24-23issuecomment-2D344959137&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=AMzd0ifqo1knKX-yHVb2L4zhGXYVmXO-yUSuGbO9d2Q&s=eAJLv-LLAj6MmNNZb01IX29LLZPrvX-3XSF6uYDrZRY&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AM-5F-2D8kI1oCHyXuKhh-5FsAYxsDV8wAzOy5ks5s3FWmgaJpZM4QaHwR&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=AMzd0ifqo1knKX-yHVb2L4zhGXYVmXO-yUSuGbO9d2Q&s=aHEmmZ6y7IaG_mpx5K6zjPyTV4eN1JsKxr7kE_hK1QI&e=.

Dr. Alan Grossfield Associate Professor Department of Biochemistry and Biophysics University of Rochester Medical Center 610 Elmwood Ave, Box 712 Rochester, NY 14642 Phone: 585 276 4193 http://membrane.urmc.rochester.edu

mangiapasta commented 6 years ago

@dmzuckerman

Good point about linear correlations. You're right, that's what I had in mind, and we shouldn't propagate the misconception that that's the only kind of correlation. I think there is a technical difference between uncorrelated and independent, since, e.g. one can probably define an uncorrelated but non-independent PDF to be the same as a product of PDFs except on a set of measure zero. I think we want to avoid measure theory here, or maybe that's the new fad in biomolecular sims...?

That being said, do anything besides linear correlations appear in Stat Mech or biomolecular sims? You mention complexity of the systems, but what predictions are you thinking of? Certainly in the UQ we're discussing, I don't think we need more than linear correlations.

Anyways, it seems like linear correlations are the relevant practical quantity that most folks will want to compute, but I appreciate your arguments about the connection (or lack thereof) between independence and correlation. I'll put in the definition of independence too.

@agrossfield @dmzuckerman Pseudo-random numbers are deterministic, so I think it's still appropriate to say the randomness we perceive is in fact a practical issue associated with chaos / complexity of the RNG.

agrossfield commented 6 years ago

@agrossfieldhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_agrossfield&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=aGE9fWeJzDpeNRjSVmxD9V0nKa2ezz-aD2ngo4y2CzM&s=3-Bqx4DMJxpJQpWXApzinppddIbSF-LMZBK3zEFOIHg&e= @dmzuckermanhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmzuckerman&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=aGE9fWeJzDpeNRjSVmxD9V0nKa2ezz-aD2ngo4y2CzM&s=VbRmhFpqs_FfUv4xqg6VhqJ2MxYwKYY1KCsrbXUP3p0&e= Pseudo-random numbers are deterministic, so I think it's still appropriate to say the randomness we perceive is in fact a practical issue associated with chaos / complexity of the RNG.

Sure — I was just chiming in to be pedantic. One of my strong points… :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmzuckerman_Sampling-2DUncertainty_issues_24-23issuecomment-2D344975471&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=aGE9fWeJzDpeNRjSVmxD9V0nKa2ezz-aD2ngo4y2CzM&s=pqkO18XGCSRf9gA8H_xNSoWvaEjWmNFrurZINZK3ALE&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AM-5F-2D8iPyupjwiQI67cHCM-5FNSXCg9nf2Eks5s3GDagaJpZM4QaHwR&d=DwMFaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=aGE9fWeJzDpeNRjSVmxD9V0nKa2ezz-aD2ngo4y2CzM&s=fwB83s7jbNdbjpFHXL7tYMjL8JhxNB_CIKM038CVsIc&e=.

Dr. Alan Grossfield Associate Professor Department of Biochemistry and Biophysics University of Rochester Medical Center 610 Elmwood Ave, Box 712 Rochester, NY 14642 Phone: 585 276 4193 http://membrane.urmc.rochester.edu

dmzuckerman commented 6 years ago

@mangiapasta @agrossfield No one is more pedantic than me ... and actually I think that word points to an 'attitude' which should guide our writing. Let's keep the main discussion to the key issues while at the same time noting subtleties and referring readers to other sources for the fine points.

Regarding (linear) correlation, we should consider carefully whether that's the essential point. After all, explaining independence in words is easy ('one variable doesn't influence the other') and that seems to be the key assumption behind general uncertainty quantification, I think. For example, in estimating std err of the mean, one requires independent samples ... I'm not sure if lack of linear correlation is sufficient. Do we really use linear correlation concepts elsewhere in the article?

@mangiapasta why don't you think about it more carefully and execute accordingly?

Thanks!!

mangiapasta commented 6 years ago

I'm like +/- 99% sure (like my use of confidence intervals :) that linear uncorrelation (pardon my abuses here) is all that is needed to compute estimates of standard uncertainty of the mean. Am I wrong on this? I've been doing a related calculation today for something with (linear) correlations, and the only terms that appear in the standard error are E[(x_i - \bar x)(x_j - \bar x)] and similar. That is, only terms quadratic terms appear when taking the expectation value of the standard uncertainty of the mean. So, linear correlations.

From: dmzuckerman notifications@github.com Sent: Thursday, November 16, 2017 1:33:51 PM To: dmzuckerman/Sampling-Uncertainty Cc: Patrone, Paul (Fed); Mention Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=2eDH6715GD2uux0j1LtVbZ2e9coaA3tYVY1G7R%2B3RTs%3D&reserved=0 @agrossfieldhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fagrossfield&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=zQanktnX%2BgS%2FeeZfJQTfkqheKfTkPRc%2Bz2xLeNRDfU4%3D&reserved=0 No one is more pedantic than me ... and actually I think that word points to an 'attitude' which should guide our writing. Let's keep the main discussion to the key issues while at the same time noting subtleties and referring readers to other sources for the fine points.

Regarding (linear) correlation, we should consider carefully whether that's the essential point. After all, explaining independence in words is easy ('one variable doesn't influence the other') and that seems to be the key assumption behind general uncertainty quantification, I think. For example, in estimating std err of the mean, one requires independent samples ... I'm not sure if lack of linear correlation is sufficient. Do we really use linear correlation concepts elsewhere in the article?

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=2eDH6715GD2uux0j1LtVbZ2e9coaA3tYVY1G7R%2B3RTs%3D&reserved=0 why don't you think about it more carefully and execute accordingly?

Thanks!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdmzuckerman%2FSampling-Uncertainty%2Fissues%2F24%23issuecomment-345015842&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=rvTnW8C57ZaoFG%2FztL3F%2BKORFNB6Gi0tD6LNXhatlto%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAd9eEf8XGovUQWMSgv-ZAF8eP1LwdgUhks5s3IAPgaJpZM4QaHwR&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=oDC%2F6H5IN3xZMxesxja4NR%2BpADFCDFwWKGq0QQZczJw%3D&reserved=0.

mangiapasta commented 6 years ago

Sorry, I should be more careful with my language.

I meant to say:

Lack of linear correlations are all that are needed to show that standard uncertainty of the mean is an unbiased estimate of the standard deviation of the mean.

From: Patrone, Paul (Fed) Sent: Thursday, November 16, 2017 1:39:07 PM To: dmzuckerman/Sampling-Uncertainty Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

I'm like +/- 99% sure (like my use of confidence intervals :) that linear uncorrelation (pardon my abuses here) is all that is needed to compute estimates of standard uncertainty of the mean. Am I wrong on this? I've been doing a related calculation today for something with (linear) correlations, and the only terms that appear in the standard error are E[(x_i - \bar x)(x_j - \bar x)] and similar. That is, only terms quadratic terms appear when taking the expectation value of the standard uncertainty of the mean. So, linear correlations.

From: dmzuckerman notifications@github.com Sent: Thursday, November 16, 2017 1:33:51 PM To: dmzuckerman/Sampling-Uncertainty Cc: Patrone, Paul (Fed); Mention Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=2eDH6715GD2uux0j1LtVbZ2e9coaA3tYVY1G7R%2B3RTs%3D&reserved=0 @agrossfieldhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fagrossfield&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=zQanktnX%2BgS%2FeeZfJQTfkqheKfTkPRc%2Bz2xLeNRDfU4%3D&reserved=0 No one is more pedantic than me ... and actually I think that word points to an 'attitude' which should guide our writing. Let's keep the main discussion to the key issues while at the same time noting subtleties and referring readers to other sources for the fine points.

Regarding (linear) correlation, we should consider carefully whether that's the essential point. After all, explaining independence in words is easy ('one variable doesn't influence the other') and that seems to be the key assumption behind general uncertainty quantification, I think. For example, in estimating std err of the mean, one requires independent samples ... I'm not sure if lack of linear correlation is sufficient. Do we really use linear correlation concepts elsewhere in the article?

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=2eDH6715GD2uux0j1LtVbZ2e9coaA3tYVY1G7R%2B3RTs%3D&reserved=0 why don't you think about it more carefully and execute accordingly?

Thanks!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdmzuckerman%2FSampling-Uncertainty%2Fissues%2F24%23issuecomment-345015842&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=rvTnW8C57ZaoFG%2FztL3F%2BKORFNB6Gi0tD6LNXhatlto%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAd9eEf8XGovUQWMSgv-ZAF8eP1LwdgUhks5s3IAPgaJpZM4QaHwR&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=oDC%2F6H5IN3xZMxesxja4NR%2BpADFCDFwWKGq0QQZczJw%3D&reserved=0.

mangiapasta commented 6 years ago

img_1917 img_1918

So, basically this derivation.

I guess technically it's for the standard variance of the mean, which, when we take the square root, gives a biased standard deviation. But lack of linear correlations is all that is needed to show that the variance is unbiased. Did I make a mistake here...?

(1917.jpg is the first page of the derivation)

From: Patrone, Paul (Fed) Sent: Thursday, November 16, 2017 1:43:56 PM To: dmzuckerman/Sampling-Uncertainty Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

Sorry, I should be more careful with my language.

I meant to say:

Lack of linear correlations are all that are needed to show that standard uncertainty of the mean is an unbiased estimate of the standard deviation of the mean.

From: Patrone, Paul (Fed) Sent: Thursday, November 16, 2017 1:39:07 PM To: dmzuckerman/Sampling-Uncertainty Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

I'm like +/- 99% sure (like my use of confidence intervals :) that linear uncorrelation (pardon my abuses here) is all that is needed to compute estimates of standard uncertainty of the mean. Am I wrong on this? I've been doing a related calculation today for something with (linear) correlations, and the only terms that appear in the standard error are E[(x_i - \bar x)(x_j - \bar x)] and similar. That is, only terms quadratic terms appear when taking the expectation value of the standard uncertainty of the mean. So, linear correlations.

From: dmzuckerman notifications@github.com Sent: Thursday, November 16, 2017 1:33:51 PM To: dmzuckerman/Sampling-Uncertainty Cc: Patrone, Paul (Fed); Mention Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=2eDH6715GD2uux0j1LtVbZ2e9coaA3tYVY1G7R%2B3RTs%3D&reserved=0 @agrossfieldhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fagrossfield&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=zQanktnX%2BgS%2FeeZfJQTfkqheKfTkPRc%2Bz2xLeNRDfU4%3D&reserved=0 No one is more pedantic than me ... and actually I think that word points to an 'attitude' which should guide our writing. Let's keep the main discussion to the key issues while at the same time noting subtleties and referring readers to other sources for the fine points.

Regarding (linear) correlation, we should consider carefully whether that's the essential point. After all, explaining independence in words is easy ('one variable doesn't influence the other') and that seems to be the key assumption behind general uncertainty quantification, I think. For example, in estimating std err of the mean, one requires independent samples ... I'm not sure if lack of linear correlation is sufficient. Do we really use linear correlation concepts elsewhere in the article?

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=2eDH6715GD2uux0j1LtVbZ2e9coaA3tYVY1G7R%2B3RTs%3D&reserved=0 why don't you think about it more carefully and execute accordingly?

Thanks!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdmzuckerman%2FSampling-Uncertainty%2Fissues%2F24%23issuecomment-345015842&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=rvTnW8C57ZaoFG%2FztL3F%2BKORFNB6Gi0tD6LNXhatlto%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAd9eEf8XGovUQWMSgv-ZAF8eP1LwdgUhks5s3IAPgaJpZM4QaHwR&data=02%7C01%7Cpaul.patrone%40nist.gov%7Cefdc27e49ed041da6e8008d52d2096a3%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636464540340664825&sdata=oDC%2F6H5IN3xZMxesxja4NR%2BpADFCDFwWKGq0QQZczJw%3D&reserved=0.

dmzuckerman commented 6 years ago

Paul, you are absolutely right that linear 'uncorrelation' is sufficient for the usual properties involving variance. I should have remembered that!

Even so, we still have the question of what to tell our readers ... my instinct is that explaining the subtle point about linear correlation will be more confusing than discussing independence which is intuitive and implies the linear uncorrelation needed for all the standard stats.

Anyway, try it how you like it as long as its acccurate, and we can read it over and see.

dwsideriusNIST commented 6 years ago

My turn to apologize for radio silence - I've been away from NIST for over a week.

Unfortunately this will take us back several days...

  1. Regarding the use of "random": I don't want to re-hash the discussion of deterministic sims versus quasi-non-deterministics sims, everything said was relevant. Perhaps we include a statement that measurements/observations in simulations are treated as "random" because of the noise induced by chaotic behavior?

  2. Regarding the "experimental" descriptor: Would it be okay to just include a statement saying that, for the present purposes, measurements and observations from simulations are experimental measurements? I'm very reluctant to not use the exact VIM terminology, because we then end up with a hybrid vocabulary that kills the point of using a standard. (I do sympathize with the point that some people, simulators included, hesitate to call simulation experiment. But for the purposes of uncertainty analysis, simulation is our experiment.)

I kind of feel like I'm clinging to the VIM... but I'd rather do that, with a few explanatory remarks, than end up with another hybrid vocabulary.

dmzuckerman commented 6 years ago

@dwsideriusNIST thanks a lot. I agree with you we should use the correct terminology, but we cannot forget that our primary goal is to persuade and educate. If our readers get confused and give up, that serves no one. So I suggest we have an informal rule that, say, once per subsection, on our first use of one of those likely-to-be-unfamiliar terms (std uncert, random variable, experimental obs, ...) we give in a parenthetical comment the common usage and refer back to the glossary.

As a hypothetical example ... A two-sided confidence interval can be estimated on the basis of the \emph{standard uncertainty,} $s$; recall $s$ is closely related to the more familiar "standard error of the mean" - see glossary.

One way or the other, we should 'pander' to our non-expert readers. They're the ones we want to adopt the best practices.

dwsideriusNIST commented 6 years ago

@dmzuckerman I'm on board with your approach - to clarify discussion with the common usage as frequent as is necessary.

One other method I'd like to use (after the paper is mostly done) is to put hyperlinks on each usage of the statistical terms that link back to the glossary. If you compile the now current repo, there is an example link on page 3 ("experimental standard deviation of the mean", in grey) that links back to page 2. The link needs to stand out more (some shade of blue?), but it gets the point across.

dmzuckerman commented 6 years ago

@mangiapasta and @dwsideriusNIST please read over the changes I pushed to the definitions section. I tried not to change any defintions, but only to improve the clarity with some added remarks. I also re-ordered (moved std uncert later) in a way that I thought promoted the logical order. I recognized that I may have made subtle errors in some wording, so please edit as you see fit.

mangiapasta commented 6 years ago

Thanks. Will take a look.

Dan S. and I have been working on an update that I expect we will push out soon. Will try to merge your changes with ours.

dwsideriusNIST commented 6 years ago

Thank you @dmzuckerman ! @mangiapasta and I will look it over and fold the changes into our version. We've actually done a major revision of that section but hadn't pushed it to the main repo yet. Based on a quick skim of your edit, our changes are in the same direction as yours.

dmzuckerman commented 6 years ago

Great. I hope my edits don't complicate yours too much.

dwsideriusNIST commented 6 years ago

@dmzuckerman, I merged the changes from @mangiapasta and me into the main repo. The major change that you'll notice is that some of the "philosophical" remarks are now in footnotes. It makes the glossary a lot less cluttered. Please let us know how it reads. Lastly, we are considering dropping "accuracy", "precision", and the independent/uncorrelated terms. Basically, unless a term is used elsewhere in the paper or is otherwise absolutely necessary, we're not keeping them in the glossary.

Also, the final subsection on "Discussion of Terminology" is toned down quite a bit. It is intended as a "why should I care about terminology" paragraph.

dmzuckerman commented 6 years ago

@dwsideriusNIST and @mangiapasta thanks very much for the revised version. I have a number of quite minor comments I hope you can address:

Would you guys be willing to look again at Sec. 7 considering two points? (1) We should adjust the notation for std uncert to be ... standard. (2) In my read, we never come out and say that the default estimate for the std uncert is the std err/exptl std dev of mean IF the measurements are independent. If you agree, would you please try to put that somewhere toward the front?

Thank you so much for all the super-careful work!!

mangiapasta commented 6 years ago

@dmzuckerman and @dwsideriusNIST

I'd like to take a crack at these changes today if it's okay. I want to edit the discussion at the end of the glossary a bit more and will take a look at the other issues that were raised. Is that okay?

dwsideriusNIST commented 6 years ago

@mangiapasta please do, as I won't be able to get to it today.

@dmzuckerman I plan to work through Sec 7 fully. I've got a start, but haven't pushed anything to the repo yet.

dmzuckerman commented 6 years ago

@mangiapasta - all yours. Let me know when you and @dwsideriusNIST are happy with both sections.

BTW - this thing is starting to look and smell like a good paper!

mangiapasta commented 6 years ago

@dwsideriusNISThttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdwsideriusnist&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=5j2alPCeo4JvnvB0C%2Fp0K4xjNUq%2BCsb2zDeRP08Atdg%3D&reserved=0 and @mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=fD%2BYvUTY%2BDxqv3JMIUPIm0YFQ262eNgBxgs4GHybfJE%3D&reserved=0 thanks very much for the revised version. I have a number of quite minor comments I hope you can address:

Response: the "d" was a typo. I removed it

I put in a remark. Let me know your thoughts.

A few changes in this regard. I demoted the independence definition to a footnote for the linearly uncorrelated data definition. Then I moved the latter above Eq. 7 and stated that the values need to be uncorrelated. Practically speaking, I don't think anyone ever tests for independence (correct me if I'm wrong though). If you agree, then I don't think independence is a useful definition. I consider it more of a mathematician's / theorist's concept, not a computational science concept. All of the UQ we do (including the autocorrelation analysis) only relies on assumptions about linear correlation.

For the time being, I commented them out, but am happy to reconsider if there is consensus to do so. I don't think we use the concepts anywhere else though, do we?

I think I fixed it, but let me know if I missed it. I'm having a hard time generating a .pdf correctly, and I don't know if I counted footnote ordering correctly.

I basically rewrote this section. Let me know thoughts. I think we need to take a harder line with why we're adopting this vocab. I don't think it's just about standardizing language, but also communicating the fact that any UQ is subjective. It's a bit pie-in-the-sky, but I think a long-term goal of this work is to induce a mindset change where people think about uncertainty more comprehensively, not just as computing error bars. To that end, changing language is one step towards that goal.

Would you guys be willing to look again at Sec. 7 considering two points? (1) We should adjust the notation for std uncert to be ... standard. (2) In my read, we never come out and say that the default estimate for the std uncert is the std err/exptl std dev of mean IF the measurements are independent. If you agree, would you please try to put that somewhere toward the front?

Thank you so much for all the super-careful work!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdmzuckerman%2FSampling-Uncertainty%2Fissues%2F24%23issuecomment-350045629&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=Fdtli8F0cVWifqWX1zAPBjLxs2ZXPLfHc5xKkG5zg5A%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAd9eEaYa1rJe-jJAA4oOIgwFzh05IYGBks5s-CcAgaJpZM4QaHwR&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=MYt%2FKcvERHhBsV3T0h8Qu4U2WVlJ9GPHyFCuRxQeTGU%3D&reserved=0.

dwsideriusNIST commented 6 years ago

@mangiapasta I found the TeX bug that was preventing compilation of the PDF. Latest commit is just a fix for that. I'm reading the revisions now.

mangiapasta commented 6 years ago

Sorry to be slow to respond to the last point.

I pasted it here for reference:

"Would you guys be willing to look again at Sec. 7 considering two points? (1) We should adjust the notation for std uncert to be ... standard. (2) In my read, we never come out and say that the default estimate for the std uncert is the std err/exptl std dev of mean IF the measurements are independent. If you agree, would you please try to put that somewhere toward the front?"

I actually don't agree that standard error / experimental standard deviation of the mean is the default estimate for uncertainty. I really think we should shy away from making this claim because that only applies to very special cases (e.g. averaging results obtained from linear analysis of multiple independent datasets). There are plenty of cases where standard error provides a biased or possibly misleading estimate of uncertainty, even when the data is independent. I added the discussion on dark uncertainty specifically because that is a tool that applies to some cases where standard error is a bad choice.

Rather, I think we need to emphasize that there isn't necessarily a standard uncertainty analysis. I'd be fine with pointing out that experimental standard deviation of the mean is a common choice, however.

From: dmzuckerman notifications@github.com Sent: Thursday, December 7, 2017 12:57:20 PM To: dmzuckerman/Sampling-Uncertainty Cc: Patrone, Paul (Fed); Mention Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

@dwsideriusNISThttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdwsideriusnist&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=5j2alPCeo4JvnvB0C%2Fp0K4xjNUq%2BCsb2zDeRP08Atdg%3D&reserved=0 and @mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=fD%2BYvUTY%2BDxqv3JMIUPIm0YFQ262eNgBxgs4GHybfJE%3D&reserved=0 thanks very much for the revised version. I have a number of quite minor comments I hope you can address:

Would you guys be willing to look again at Sec. 7 considering two points? (1) We should adjust the notation for std uncert to be ... standard. (2) In my read, we never come out and say that the default estimate for the std uncert is the std err/exptl std dev of mean IF the measurements are independent. If you agree, would you please try to put that somewhere toward the front?

Thank you so much for all the super-careful work!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdmzuckerman%2FSampling-Uncertainty%2Fissues%2F24%23issuecomment-350045629&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=Fdtli8F0cVWifqWX1zAPBjLxs2ZXPLfHc5xKkG5zg5A%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAd9eEaYa1rJe-jJAA4oOIgwFzh05IYGBks5s-CcAgaJpZM4QaHwR&data=02%7C01%7Cpaul.patrone%40nist.gov%7C03f46da886074577dbde08d53d9bf6dc%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636482662438224767&sdata=MYt%2FKcvERHhBsV3T0h8Qu4U2WVlJ9GPHyFCuRxQeTGU%3D&reserved=0.

dmzuckerman commented 6 years ago

@mangiapasta thanks for that comment. I guess the question becomes, what do we recommend for the nonexpert readers (our most important audience, I think)? They need something concrete. I think we need to tell them to do (a), (b), or (c) under suitable conditions and clearly explain the conditions and pitfalls.

You wrote, "There are plenty of cases where standard error provides a biased or possibly misleading estimate of uncertainty, even when the data is independent." Since I don't think I have a good grasp on when the standard error is not a good uncertainty measure, it would be very helpful if you could write up something with clear logic and examples. I will be happy to help edit, once I can understand what you mean.

I still think Sec. 7 is the most important whether we like it or not ... people know they need error bars for publication and they would like very explicit guidance.

Once you and @dwsideriusNIST are happy with it, let me know and I'll get to work on it.

mangiapasta commented 6 years ago

I'm in the process of writing up a short example, but it will take a few days. I'm away at a conference.

From: dmzuckerman notifications@github.com Sent: Monday, December 11, 2017 3:56:37 PM To: dmzuckerman/Sampling-Uncertainty Cc: Patrone, Paul (Fed); Mention Subject: Re: [dmzuckerman/Sampling-Uncertainty] Use of "random" in the glossary (#24)

@mangiapastahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmangiapasta&data=02%7C01%7Cpaul.patrone%40nist.gov%7C7eeeec4dee5f4610310508d540d9ac81%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636486225995157133&sdata=7LZq%2BKisPwl6IVIm8gdhPlaz%2FunoQrCCbytjzF0vtA4%3D&reserved=0 thanks for that comment. I guess the question becomes, what do we recommend for the nonexpert readers (our most important audience, I think)? They need something concrete. I think we need to tell them to do (a), (b), or (c) under suitable conditions and clearly explain the conditions and pitfalls.

You wrote, "There are plenty of cases where standard error provides a biased or possibly misleading estimate of uncertainty, even when the data is independent." Since I don't think I have a good grasp on when the standard error is not a good uncertainty measure, it would be very helpful if you could write up something with clear logic and examples. I will be happy to help edit, once I can understand what you mean.

I still think Sec. 7 is the most important whether we like it or not ... people know they need error bars for publication and they would like very explicit guidance.

Once you and @dwsideriusNISThttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdwsideriusnist&data=02%7C01%7Cpaul.patrone%40nist.gov%7C7eeeec4dee5f4610310508d540d9ac81%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636486225995157133&sdata=p5DP6G8zNlj6B1iEo5ZRlY7Vnob3BD%2BD9%2BmBKkA9znY%3D&reserved=0 are happy with it, let me know and I'll get to work on it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdmzuckerman%2FSampling-Uncertainty%2Fissues%2F24%23issuecomment-350855580&data=02%7C01%7Cpaul.patrone%40nist.gov%7C7eeeec4dee5f4610310508d540d9ac81%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636486225995157133&sdata=Pp5abklrCnVp0Ko7oPt%2FmX85j82qef%2FhCOWUBf47nwE%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAd9eESawZAwlVt60gH7vMWDAvkUyOr9oks5s_ZcFgaJpZM4QaHwR&data=02%7C01%7Cpaul.patrone%40nist.gov%7C7eeeec4dee5f4610310508d540d9ac81%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C636486225995157133&sdata=utM58M%2BuTphohagQHTHNj9u896gmIzzulShf%2FA0r%2BJM%3D&reserved=0.

mangiapasta commented 6 years ago

@dmzuckerman A relatively easy example that comes to mind is the case where we need to average simulated results that are independent but not identically distributed. I attached some notes to show how a weighted average provides a better estimate of the true mean. The corresponding "standard error" is not computed in the usual way, but rather in terms of a harmonic average of the uncertainties associated with each realization of the random variable. I run into this case and generalizations thereof somewhat frequently.

Along these lines, the dark uncertainty analysis takes the notion of a weighted mean a step further by postulating that the random variables have both a common and realization-dependent component to their distributions. The former is the "dark" uncertainty that can only be estimated after the realization-dependent uncertainties have been estimated.

KIC Document 0001 (2).pdf

dmzuckerman commented 6 years ago

I added to the remark after Eq (5) because I think it's confusing to novices why a linear average of simulated/experimental data is the same as weighted avg defn (1).

dmzuckerman commented 6 years ago

@dwsideriusNIST and @mangiapasta I think you guys did a great job with the defns section, and the closing paragraphs are particularly nice.

dmzuckerman commented 6 years ago

@dwsideriusNIST and @mangiapasta - please see a few nit-picky comments/questions embedded in the text of the defns section in my recent commit.