Quantify alpha recovery success

[LaurencePeanuts]

We should test the robustness of the alpha (the prior normalization) recovery when we maximize the evidence. To do so, we should generate many mocks (each with their own noise realization) with alpha = 1, and check the distribution of the recovered alpha values. This would allow us to check that our reconstruction method doesn't introduce biases in alpha.

One possible measure of robustness is to calculate

[LaurencePeanuts]

I tried this with l_max = 8, l_min = 2, n_max = 2, n_min = 1, for 10 000 mock data sets.

Here is the histogram of the recovered values of alpha, with the best-fit normal distribution overlaid:

For those 10 000 samples, the equation given above gave 0.0234348783315.

[drphilmarshall]

Sweet! Does this mean you have also solved the problems of the units/meaning of the normalisation/regularisation parameter? This bodes really well for the inflation inference :-)

On Tue, Mar 1, 2016 at 3:10 PM, Laurence Perreault Levasseur < notifications@github.com> wrote:

I tried this with l_max = 8, l_min = 2, n_max = 2, n_min = 1, for 10 000 mock data sets.

Here is the histogram of the recovered values of alpha, with the best-fit normal distribution overlaid:

[image: alphahist_lmax8_nmax2] https://cloud.githubusercontent.com/assets/13594101/13445118/3f33a192-dfbf-11e5-88fa-913ddc0a807f.png

For those 10 000 samples, the equation given above gave 0.0234348783315.

— Reply to this email directly or view it on GitHub https://github.com/rogerblandford/Music/issues/30#issuecomment-190955116 .

[LaurencePeanuts]

No, that's not what it means... I make the mock data from my own power spectrum, so I'm generating the mock with alpha = 1. For this, Issue #23 only cause the noise to be bigger (or smaller, but here it's slightly bigger) relative to the CMB perturbations than it should be if I had used the proper power spectrum normalization to start with. The plot above just shows that the recovered alpha isn't biased (best-fit mean of 1.02 vs true value of 1), and how precise the recovery is.

[drphilmarshall]

I see - so #23 is still open. But my last point still stands - this is good news for the inflation inference!

On Tue, Mar 1, 2016 at 4:24 PM, Laurence Perreault Levasseur < notifications@github.com> wrote:

No, that's not what it means... I make the mock data from my own power spectrum, so I'm generating the mock with alpha = 1. For this, Issue #23 https://github.com/rogerblandford/Music/issues/23 only cause the noise to be bigger (or smaller, but here it's slightly bigger) relative to the CMB perturbations than it should be if I had used the proper power spectrum normalization to start with. The plot above just shows that the recovered alpha isn't biased (best-fit mean of 1.02 vs true value of 1), and how precise the recovery is.

— Reply to this email directly or view it on GitHub https://github.com/rogerblandford/Music/issues/30#issuecomment-190986076 .

[LaurencePeanuts]

I filled up a table quantifying the robustness of the alpha recovery for various l_max and n_max here. There seems to be some systematic bias, in the sense that the recovered alpha is systematically above the input value of 1, but it behaves better and better as n-max becomes bigger. I calculated the values in the table from 1000 realizations, and I didn't go to higher n_max because things started to be pretty slow at around n_max=6.