Sigma intrinsic and new whitening method summary

[coveralls]

Coverage decreased (-2.4%) to 13.604% when pulling 3d529c2d25b9c8c079aac977c850c0ad4ebc2c12 on chto:add_sigma_intrinsic_and_new_whitening into f7f37a1a97cbd16e31363fc8baf076525c9bc280 on DarkEnergyScienceCollaboration:master.

[chto]

Hi, @drphilmarshall

In this branch, I add two notebooks which are Sigma_Intrinsic_Analysis.ipynb and SLTimer_TDC2_NewWhitening.ipynb.

In Sigma_Intrinsic_Analysis.ipynb: I do an analysis on how sigma_intrisic affect the likelihood. In SLTimer_TDC2_NewWhitening.ipynb: I do the same analysis as in Sigma_Intrinsic_Analysis.ipynb, but change the whitening method. The likelihood for new whitening method is strange and I am still finding the reason. But I think it is not likely to be caused by some bugs in the code, after checking the code again and again. Therefore, I think it is a good time to let you know.

If you have any suggestion or comments, please let me know. Thanks.

[drphilmarshall]

@chto Great work! It's hard to comment inline on notebooks, so I'll do it here instead.

Sigma_Intrinsic_Analysis.ipynb

• Your plots show very nicely how the inclusion of the $\sigma_{intrinsic}$ parameter makes a huge difference to the log likelihood of the parameters given the data.
• Look how sharp the peak in likelihood is with respect to $\sigma_{intrinsic}$ - you had to take logs twice! This is what I meant by "there's always more information in the data about the noise than there is about the signal". The sharpness of this peak is a potential problem, if its not sharp for every value of $\Delta t$ - but I suspect it is. Because the likelihood for $\sigma{intrinsic}$ is so sharply peaked, a good strategy will be to first optimize $\sigma{intrinsic}$, and then sample close to this value and across the prior range for $\Delta t$. Can you try adding a few more curves to your plot of the log(log(likelihood)) for $\sigma{intrinsic}$ at fixed $\Delta t$, please? If you consistently find the same optimal $\sigma{intrinsic}$ no matter what $\Delta t$ you are at, you can just pick one $\Delta t$ (nominally zero) and first estimate $\sigma_{intrinsic}$ that way.
• The plot of log L vs $\Delta t$ at optimal $\sigma_{intrinsic}$ is interesting. Can you please try exponentiating all of log L, log prior and log posterior, and overlaying them on the same graph? (You'll need to manually renormalize L to unit integral for a good comparison). I want to see whether we have multiple peaks close to the true value (as well as far away from it).
• The conclusion about log L not returning the true time delay when $\sigma_{intrinsic}$ is far from optimal is a good lesson; it's related to the way your plots show that even when $\sigma{intrinsic}$ is optimal_ there may still be times when maximizing the log L returns the wrong time delay. This is basically because while our new model is better, it's still wrong! Welcome to observational cosmology: all our models are wrong, but we care deeply about the cosmological (hyper-)parameters behind them. Every cosmological probe suffers from this issue: our task (in general) is to mitigate against this problem however we can. Ideas include: make better models; test against simulations to see how bad our models are; design robust data summaries that can be shown (with simulations) to reduce the impact or avoid entirely having a bad model; and so on.

[drphilmarshall]

SLTimer_TDC2_NewWhitening.ipynb

• This whitening looks to be doing much better! The flux error bars are unchanged when seasonal=True, so I wonder: can we compare the absolute value of the likelihood in this analysis with that in your other Sigma_Intrinsic_Analysis.ipynb notebook, where seasonal=False? I think we might be able to. With new whitening, the log L values are around +1000, whereas in the other notebook they are more like -300 - so it does seem like the new whitening is a better model. You should think about whether or not this is a fair comparison though.
• I think you should make the same plot showing prior, likelihood (renormalized) and posterior for $\Delta t$ at optimal $\sigma_{intrinsic}$ that I suggested above. I bet the posterior is a bit different to the prior, but it's just hard to see in the logarithmic scale plots. I agree that the log likelihood is strange, but it is mostly flat (on the scale of the prior) rather than weirdly peaked. So if there is an issue it might be that the whitening is somehow removing the signal we are looking for. Maybe seasonal is also the wrong approach?

[drphilmarshall]

OK, all these commits were also in your issue/42/initialization branch, which I just merged into issue/40/tdc2gatewaytest. Can you please pull this branch from the base repo and check that everything looks right to be your current version? Any problems, please issue them. Thanks!