Validate Fisher results

[dlanzieri]

This issue is to track the steps we need compute in order to be sure that we can trust our Fisher matrix. As discussed before, the steps to follow are:

• [x] Determine how many samples we need to be able to trust our Fisher matrix
• [ ] As soon as we trust the matrix, see what the contours look like for PS and peaks when all params are included
• [ ] See what the contours look like with just Omega_c and sigma8
• [ ] Based on the previous results, understand which degree of freedom leads to an important change in the behaviour of the constraints in Omega_c and sigma8

[dlanzieri]

@EiffL Concerning the first task, the following image shows the overlapped Fisher contours made by averaging over a different number of samples. The red contours are realized using all the samples (170 jacobians), the violet contours just 34. What I see from this plot it's that the contours became larger when I use a larger sample for the average. In the following plot I'm only showing the contours realized with : RED: 170 jacobians YELLOW : 136 jacobians GREEN: 102 jacobians It seems that between 102 and 136 samples the contours are still differents, but between 136 and 170, they start to be more similars and stables. Here, I just keep: RED: 170 jacobians YELLOW : 136 jacobians

[dlanzieri]

@EiffL This is what I was thinking to do: I would increase a bit the sample used to compute the average (at least for the power spectrum and in this validation stage). Then, If we see that this trend is confirmed and that with ~180/200 objects we can trust of our Fisher matrix, I will use this number for the further analysis (peak counts, l1norm, IA etc). What do you think?

[EiffL]

Ah very nice! To make it a round number we could say 200 samples? to ensure the jacobian is stable.

[EiffL]

Note that you would then want to do a similar check for your other stastistics. Because it all depends on how noisy a given bin is. Hopefully 200 will be a good number for all of them

[dlanzieri]

@EiffL I increased the number of samples to average the Ps-Jacobian. I made a plot of the mean Ps-Jacobian computed with different numbers of samples (normalized by the square root of the diagonal of the covariance matrix). You can find in this notebook the plots for more different number of samples, for an easier visualization here I show only 3 comparisons. The Jacobians seem still noisy, especially for sigma8, h and ns. and in the following plot I'm only showing the contours realized with : RED: 250 jacobians Violet: 230 jacobians Orange: 201 jacobians Then, I made a similar check also for the peak counts. Fortunately, the jacobians seem less noisy and much more stables (slightly noisy h and ns) and in the following plot I'm only showing the contours realized with : RED: 160 jacobians Blue : 230 jacobians Yellow: 200 jacobians For both the statistics the contours don't seem change their orientations too much with a number of samples higher than ~160 (in the notebook you can find other 2 plots with more number of sample, and there the change of orientation is more evident). However, the width of the contours change for both statistics with a different number of samples.

[EiffL]

This is great Denise! So, yeah Fisher matrices are always tricky :-/ but I think this is not too bad. It looks "fairly stable" and we have an idea of how much of an error to expect on the contours, so we know if when comparing 2 different stats we should read too much into some differences.

[dlanzieri]

@EiffL We have also discussed the importance of making the power spectrum analysis consistent with the peak counts analysis in order to compare their sensitivity. If we apply 1 arcmin smoothing the plot of the angular power spectrum looks like this: As we already noted, the effect of smoothing starts to suppress the information for ell around 10000, but we consider the power spectrum only in a range between 300 and 3000. If we wanted to apply a smoothing so that the angular power spectrum lost power for ell = 3000, we should apply a higher smoothing. This for example corresponds to 5 arcmin: I was wondering if we should increase the smoothing or consider the power spectrum until ell~ 10000.

[EiffL]

You are correct, this 5 arcmin smoothing seems more consistent. For an LSST Y1 this is probably around the scales we would use. In recent DES papers they went to 7.5 arcmin https://arxiv.org/pdf/2110.10135.pdf

[EiffL]

hummm but can you convert the l=3000 to an arcmin scale? what value do you get?

[dlanzieri]

l=3000 should be ~ 3.6