search

reflectivity / reproducibility

Reproducible research using Neutron and X-ray Reflectometry
https://www.reflectometry.org/
3 stars 1 forks source link

Define some statistical method for the comparison of two experimental datasets #18

Open arm61 opened 4 years ago

arm61 commented 4 years ago

We need a way to compare "identical" datasets.

arm61 commented 4 years ago

WIP here https://github.com/reflectivity/reproducibility/blob/master/comparing_datasets.md

jfkcooper commented 3 years ago

As detailed in https://arxiv.org/pdf/2103.08973.pdf Section 3.1 if the points are indeed at the same place (a rebin or similar required here though). Then the Hotelling t^2 test can give the probability that the datasets are different (and the significance of that difference). The inverse way of thinking about the problem, but having looked into this quite alot, I couldn't find anything better.

arm61 commented 3 years ago

What about if the q-ranges are different? I guess we only use overlapping regions then.

jfkcooper commented 3 years ago

Pretty much all you can do I think.

jfkcooper commented 3 years ago

Have made a thing (mostly stolen from James Durant's code), it will do this and is in the main repo as am not sure where sripts should live

jfkcooper commented 2 years ago

I think I need to do some more work on this still as am not sure it is correct yet...

arm61 commented 2 years ago

I quite liked the use of a Hotelling r2 test, what is wrong with it?

jfkcooper commented 2 years ago

I think the principle might be okay, but my implementation isn't quite right. The current "solution" works in count space (because we need to somehow include the error bars of the points), however this makes something with a scalefactor of a half which has been counted twice as long equal to a dataset of unity scalefactor counted half as long, i.e. not what we want really. I had thought it was working but recently revisited it and found it wasn't. The Hotelling test might be helpful still, but I think it needs a different way to include the error bars of the datapoints

wpotrzebowski commented 2 years ago

Not sure if it is useful for your case but here is a method applied to Small Angle Scattering data "for assessing differences between one-dimensional spectra independently of explicit error estimates" https://pubmed.ncbi.nlm.nih.gov/25849637/. There is an open-source implementation of the method in freesas (https://github.com/kif/freesas/blob/master/freesas/app/cormap.py)