search

OxIonics / ionics_fits

Small python fitting library with an emphasis on Atomic Molecular and Optical Physics
Apache License 2.0
0 stars 0 forks source link

Error bars for binomial fitter when close to extrema #180

Closed r-srinivas closed 3 months ago

r-srinivas commented 3 months ago

Ran into an issue with the error bars here

https://github.com/OxIonics/analysis/blob/master/notebooks/2024.03.27/Figuring%20out%20discrepancy%20in%20uncertainty.py

Seems to be an issue when the points are close to the extrema, which is something we face when our parity is close to 1.

hartytp commented 3 months ago

I looked into this a bit. It's an interesting test case. Essentially, fitting a SineMinMax with the min set to 1e-6 and 10,000 shots per point. Moreover, the way the x-axis and phases have worked out there is a point at exactly the bottom of the fringe. In this situation, the best fit to the data will essentially always put the minimum at exactly 0.

So if we plot histograms of the fits, that's what we'll get. However, we don't actually have enough information to reliably pin the fit down to this level, which is why the error bars are higher.

NB given how close to the lower bound the parameter value is, this was always going to be a case where the distribution of fitted parameters is highly non-normal, so error bars don't have an obviously intuitive meaning.

hartytp commented 3 months ago

i.e. this is an artefact of the fact that we're sampling a discrete distribution too few times to build up an accurate picture of it - if we take a Binomial distribution with p = 1e-6 and sample 1e4 times, we will get k=0 successes 99% of the time. Fitting to those just tells us p=0. Looking at the variance of those fits overstates our knowledge of the true probability distribution.