search

simonsobs / map_based_simulations

Map based simulations for the Simons Observatory
4 stars 1 forks source link

Low ell filtering for noise spectra #29

Closed zonca closed 4 years ago

zonca commented 4 years ago

We have new noise spectra from https://github.com/simonsobs/so_noise_models

New noise spectra plots

image image

You can run the code in the demos/ folder to reproduce the plots and play with the curves.

The main difference is a fix in the low ell contribution from the atmosphere that should be now higher at high frequencies and lower at low frequencies.

For the last noise simulations run, @thibautlouis @dpole reported we had too much low-ell noise, see https://github.com/simonsobs/map_based_simulations/issues/14 and I re-ran a new version of the sims with zeroed noise for ell < 30, https://github.com/simonsobs/map_based_simulations/tree/master/201906_noise_no_lowell

Is it now clear what was the cause of that? What should we do for the next noise run which in now in preparation (#21) ?

@thibautlouis @dpole @jcolinhill @msyriac @keskitalo @mhasself

dpole commented 4 years ago

As far as I am concerned, the noise was not higher than it should be. The problem is that this spectrum diverges at the very large scales and this creates several difficulties, despite we typically ignore those scales.

A practical thing, which I nevertheless find relevant, is that the amplitude of those very long modes dominates any other signal. This makes impossible any visual inspection of noisy maps.

A more scientific argument can be made about, e.g., power spectrum estimation (@thibautlouis please chime in to confirm/correct my statement). The mode-mixing matrix between the largest scales and large/intermediate scales is not zero. When the matrix is binned, inverted and applied to pseudo-Cls, the l < 30 power leaks in the estimation of large/intermediate scales, decreasing numerical precision and increasing cosmic variance also in the l>30 scales. This happens because the fraction of leaked power may be small, but the divergence makes the power large in absolute terms.

The point about removing modes at ell < 30 is that we will probably not have them in the maps. Filter-and-bin map-makers will suppress that power because of some high-pass filter in the TOD processing. Maximum likelihood map-makers will try to restore some of the suppressed power but probably will not converge at those scales, resulting in a lack of power at very large scales.

Where to set the threshold in ell is arbitrary at this stage, but probably the same filter should be applied consistently to signal maps. What do people think?

thibautlouis commented 4 years ago

I agree with @dpole, and yes ideally we should also remove the mode in the signal map.

zonca commented 4 years ago

however in signal simulations having power at low-ell is less of a problem because the are full sky and you can remove them later on, right? Anyway I'll keep this in mind and ask feedback again when we will be working on next signal release.

About noise, I wonder if we should ask @keskitalo to run some time domain simulations to estimate a transfer function we can then apply to the map-based simulations. Currently I think this might not be needed as you are not much interested in those scales anyway. But if anyone has a different opinion please let us know.

zonca commented 4 years ago

from discussion at today's TOD2Maps call, @mhasself will implement a low-ell damping directly into the noise spectra in so_noise_models, current plan is exponential curve that starts at ell=60. Will update here when we have more news.

zonca commented 4 years ago

this is implemented in so_noise_models, see the rolloff_ell argument. https://github.com/simonsobs/so_noise_models/blob/master/so_models_v3/SO_Noise_Calculator_Public_v3_1_1.py#L261