PySM v3.4b6 synchrotron models s4, s5 and s7 validation

[1cosmologist]

I did a validation study of the s4, s5 and s7 synchrotron models in PySM v3.4b6 by comparing bandpass integrated LFI 30 GHz and WMAP K band synchrotron simulations with observations.

Bandpasses used: LFI RIMO 30GHz average bandpass, WMAP 9yr bandpass frequency responses (K11, K12)

Observations used: NPIPE (PR4) map full mission and splits A/B at NSIDE=1024, and WMAP 9yr K band maap at NSIDE=512. SMICA 2018 (PR3) CMB map to remove the CMB at map level (for some comparisons). Planck full EE power spectrum for removing CMB EE power contribution to LFI 30 GHz data. Planck polarized point source mask for 30 GHz.

Power spectra comparison: The IQU maps are masked with the apodized polarized point source mask. This masked map is used to obtain E and B mode maps. I mask the E/B maps with apodized 70%, 60% or 50% sky fraction masks devised based on brightness of the LFI 30 GHz polarized intensity maps. I compute power spectrum with anafast and divide by the fsky. For actual LFI/WMAP data we also correct for the CMB contribution to the power.

Note: The power spectra for both the data and the simulations are processed with the same pipeline.

Here is the result for 70% sky for LFI x WMAP case.

A summary of the results can be found in the slides linked here. The jupyter notebook used in this validation study can be found here

[giuspugl]

thanks a lot @1cosmologist for doing this test.

I compute power spectrum with anafast and divide by the fsky. For actual LFI/WMAP data we also correct for the CMB contribution to the power.

@zonca and I actually estimated the power spectra with namaster to correct to mask leakage from the Planck GAL0* masks. Your masks slightly differ from ours and this represents a valuable test to further validate the maps. Do you mind to run namaster and see how the spectra look like ? (you can find the routines in the linked notebooks in the pysm doc, see e.g. https://pysm3.readthedocs.io/en/latest/preprocess-templates/small_scale_dust_pysm3.html .

[brandonshensley]

@1cosmologist: thanks for looking into this. However, I'm having a hard time following your notebook. Could you please use PySM directly to make maps? Currently you write:

synch_template, header = hp.read_map('../data/synch_template_nside2048.fits', field=None, h=True)

But this is not reproducible and your maps do not look like the current PySM maps. Let me know if you need help generating full sky maps at the desired NSIDE using PySM directly. Thanks.

[1cosmologist]

@brandonshensley The s4, s5, s7 maps are produced with PySM directly. These are produced by codes in cells 5 and 9. Because they looked bit strange, we actually checked the synchrotron template being used in the models to be sure that we are not producing it. All results are computed with maps produced in cells 5 and 9. They were saved and read for subsequent runs. Let me know if I made any mistake with the code to produce the maps. Thanks for going through the code. I can update the notebook with more comments if it's too messy.

[seclark]

A simplified version of this notebook with more comments and just the "minimum working example" version of the problem here would be immensely helpful. Thanks!

[brandonshensley]

Agreed. Even just my 30 GHz intensity map (below) looks very different than those plotted in the notebook.

[brandonshensley]

Ok folks, hang tight, it looks like there might be a dependency issue where the same PySM version is yielding different output... @NicolettaK and I ran the same code, she got plots that looked like @1cosmologist's, I got things that looked more as expected. Maybe healpy version? I am using 1.16.2.

[brandonshensley]

Here is my notebook, where the synchrotron model looks to be behaving as expected.

[NicolettaK]

And this is mine. I downloaded the file synch_template_nside2048.fits from https://portal.nersc.gov/project/cmb/pysm-data/synch/

[brandonshensley]

@zonca , can you take a look at Nicoletta's and my notebooks to see if you have an idea of what is going on? Thanks!

[zonca]

@zonca , can you take a look at Nicoletta's and my notebooks to see if you have an idea of what is going on? Thanks!

yes, sure

[1cosmologist]

@brandonshensley @NicolettaK Here is a simplified notebook computing synchrotron maps at NSIDE=1024 with and without bandpass integration for LFI 30 GHz. Both maps look like my maps from previous notebook.

I have also shown the versions of the main packages used in the notebook, in case it is a dependency issue. Let us first agree if these maps look like what we should get before I compare with data.

[zonca]

I confirm there is a bug in the generation of templates for beta6, I am tracking it down.

[zonca]

@brandonshensley I think you might have the template for beta4 cached on your machine. Try:

rm -r ~/.astropy/cache

[brandonshensley]

@brandonshensley I think you might have the template for beta4 cached on your machine. Try:

rm -r ~/.astropy/cache

Thanks, that "fixed" it!

[zonca]

the bug is related to https://github.com/galsci/pysm/issues/151:

the modulation map for the temperature of synchrotron is between 2 and 3, however, in the notebook that generates the templates for beta6, it is normalized on the fly. Also the polarization one has some tweaking on the fly. See cell 59 of the relevant notebook. So the version I saved and used to generate the templates is unnormalized and boosts the injected small scales abnormally.

I didn't catch the bug because I was rushing and completely forgot to compare my outputs to @giuspugl 's as I did for dust in https://github.com/galsci/pysm/issues/148.

Thanks @1cosmologist for pointing out the problem.

Now the matter is, do we have anyone volunteering to tell Julian?!

[zonca]

I'll:

• [x] fix the bug
• [x] regenerate all the templates for synchrotron
• [x] compare with @giuspugl's
• [x] release beta7
• [x] check the impact on CMB-S4 simulations

[seclark]

Thank you very much @zonca ! And @1cosmologist thank you for finding this: I hope you can run the corrected maps through your pipeline as additional validation.

[zonca]

bug fix in https://github.com/galsci/pysm/pull/154

[zonca]

Here is the plot that shows better the issue in beta6, i.e. the excess of small scales, notebook:

This is the same for the upcoming beta7, notebook:

[zonca]

ok, I regenerated the templates after fixing the bug. Now I am using a date in the filename of the templates, so the caching issue @brandonshensley experienced shouldn't happen anymore. I changed the location of the old templates, so that people do not use them by mistake. At this point beta6 should not work anymore, on purpose.

I have released 3.4.0b7. @1cosmologist can you please rerun your checks? In the meantime I am rerunning the CMB-S4 map based simulations affected by this issue.

[1cosmologist]

The beta 7 maps look a lot better:

Running the full set of tests on the beta 7 maps now. I will update once complete.

[1cosmologist]

Beta 7 brings models much closer to the observations:

There is perhaps a small excess of power in the models. The full results summary for the beta 7 synchrotron maps can be found here. The updated Jupyter notebook is available here.

[brandonshensley]

Excellent news. Thank you for turning this around so quickly @zonca and @1cosmologist!

[delabrou]

Hi everyone, we have a traceability problem... we need to have some versioning for the PySM input maps, so that one cannot run the model thinking they use the most up-to-date model while in fact using a previous map stored in the cache... Can this be fixed easily?

[zonca]

@delabrou yes, this is a problem that only affects people that have used a beta release. People using stable releases are not affected. I plan to rename all maps and add a timestamp when I release the stable 3.4.0. So that even if people used a beta release previously, they would not by mistake use an outdated file from the cache.

[1cosmologist]

Further validation of the synchrotron models in the CMB-S4 deep patch: Looks pretty good! Slides updated with summary of results for the CMB-S4 patch.

[zonca]

thanks @1cosmologist, could you please share the latest version of the notebook? It would be nice to add it to the documentation of PySM

[1cosmologist]

The notebook for the CMB-S4 SAT-TMA common patch analysis can be found here.

[zonca]

this analysis is included in the paper in preparation, no need to add it to the documentation of PySM.