PySM prerelease with GNILC models d9,d10,d11 and synchrotron models s4,s5,s6,s7

zonca commented 2 years ago

I've published to PyPI PySM 3.4.0 beta 1, you can install it with:

pip install --pre pysm3

(--pre grabs pre-releases)

Alternatively you can test in your browser using Google Colab (or use the notebook code locally):

https://colab.research.google.com/drive/1LI7zGC2Exd8R0HiYMl6b3Bnw-0bRp0Ws?usp=sharing

Please report any bugs here or open dedicated issue. Feedback on the docs is particualrly appreciated, see https://pysm3.readthedocs.io/en/latest/models.html#dust, you can also click on "Edit this file on Github" to create a pull request directly.

zonca commented 2 years ago

there is currently a bug in CO #113 , it is fixed in the main branch #114 , as soon as I have a first version of synchrotron in a few days I'll make another beta release with this fix.

zonca commented 2 years ago

I just released 3.4.0b2, this release has:

fixed the bug I recently introduced into the CO lines model, see #114, thanks @mousset for reporting it
implementation of the Synchrotron models, see docs in https://pysm3.readthedocs.io/en/latest/models.html#synchrotron

You can install it with:

pip install --pre pysm3 --upgrade

Or checkout the Colab link above.

zonca commented 2 years ago

New PySM 3 models included:

s4, s5, s6 Synchrotron
co1, co2, co3 CO lines
d9, d10, d11 GNILC Dust
d12 MKD 3D Dust

zonca commented 2 years ago

Release 3.4.0b3 which also includes s7, synchrotron model with curvature. I haven't implemented a stochastic model for synchrotron with curvature, please let me know if anyone is interested in that.

zonca commented 2 years ago

@brandonshensley @seclark @giuspugl @bthorne93 I prepared a fireslide for the CMB-S4 spring meeting with quick tour of the new models, please provide feedback: https://docs.google.com/presentation/d/1_JkiKpVvbCL4lMUPWonDszq4mR8S6XWnTmN6JHdh-v4/edit?usp=sharing

brandonshensley commented 2 years ago

@brandonshensley @seclark @giuspugl @bthorne93 I prepared a fireslide for the CMB-S4 spring meeting with quick tour of the new models, please provide feedback: https://docs.google.com/presentation/d/1_JkiKpVvbCL4lMUPWonDszq4mR8S6XWnTmN6JHdh-v4/edit?usp=sharing

This looks great to me! Might be useful to give a reference (hyperlink/arXiv number/journal citation) for the MKD model. Susan and I have a talk on Friday where we can explain in a little more detail.

zonca commented 2 years ago

PySM 3.4.0b4

We have a new beta, this has been already used for production runs for CMB-S4, and will soon for Litebird and SO.

All the models have changed, due to changing how PySM does the spherical harmonics transform, on top of that GNILC and synchrotron have brand new small scales (thanks @giuspugl) and we have Websky extragalactic (thanks @xzackli)

List of the models:

s4, s5, s6, s7 Synchrotron
co1, co2, co3 CO lines
d9, d10, d11 GNILC Dust
d12 MKD 3D Dust
cib1, rg1, ksz1, tsz1 Websky 0.4 components (rg is Radio Galaxies)
c3, c4, Unlensed and Lensed Websky CIB components

As usual, update with:

pip install --pre pysm3 --upgrade

See https://github.com/galsci/pysm/releases/tag/3.4.0b4

zonca commented 1 year ago

PySM 3.4.0b5

This new beta release fixes the color correction in the MKD dust model (d12) See #141

https://pypi.org/project/pysm3/3.4.0b5/

zonca commented 1 year ago

PySM 3.4.0b6

New implementation of the small scales for Dust and Synchrotron https://github.com/galsci/pysm/issues/148

Need to cleanup documentation, then hopefully this will become PySM 3.4.0

zonca commented 1 year ago

PySM 3.4.0b7

Bugfix on Synchrotron #153

zonca commented 1 year ago

PySM 3.4.0b9

Releasing 3.4.0b9, the most important improvement is a bugfix in the dust beta and Td I moved the broken templates out of the way and marked the fixed templates with a datestamp. So old PySM version will stop working to avoid people running broken models by mistake.

https://pypi.org/project/pysm3/3.4.0b9/

zonca commented 1 year ago

Logs of unit tests of pysm 3.4.0b9 running at NERSC:

(pycmb) zonca@login32 pysm (main) $ pytest -v
======================================================= test session starts =======================================================
platform linux -- Python 3.10.0, pytest-7.2.0, pluggy-1.0.0 -- /global/common/software/cmb/zonca/conda/pycmb/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/global/u2/z/zonca/p/software/pysm/.hypothesis/examples')
rootdir: /global/u2/z/zonca/p/software/pysm, configfile: pyproject.toml
plugins: arraydiff-0.5.0, filter-subpackage-0.1.1, openfiles-0.5.0, hypothesis-6.56.4, remotedata-0.3.3, cython-0.2.0, doctestplus-0.12.1, anyio-3.6.2, astropy-0.10.0, cov-4.0.0, astropy-header-0.2.2, nbval-0.9.6, mock-3.10.0
collected 104 items / 2 skipped                                                                                                   

pysm3/tests/test_ame.py::test_model[a1-30] PASSED                                                                           [  0%]
pysm3/tests/test_ame.py::test_model[a1-100] PASSED                                                                          [  1%]
pysm3/tests/test_ame2.py::test_model[a2-30] PASSED                                                                          [  2%]
pysm3/tests/test_ame2.py::test_model[a2-100] PASSED                                                                         [  3%]
pysm3/tests/test_bandpass_convert_units.py::test_Bandpass_Unit_Conversion::test_bandpass_unit_conversion_CMB2MJysr PASSED   [  4%]
pysm3/tests/test_bandpass_convert_units.py::test_Bandpass_Unit_Conversion::test_bandpass_unit_conversion_MJysr2KRJ PASSED   [  5%]
pysm3/tests/test_bandpass_convert_units.py::test_bandpass_convert_integration::test_bandpass_convert_units PASSED           [  6%]
pysm3/tests/test_cmb.py::test_cmb_map PASSED                                                                                [  7%]
pysm3/tests/test_cmb.py::test_cmb_map_bandpass PASSED                                                                       [  8%]
pysm3/tests/test_cmb.py::test_cmb_lensed[c1-30] PASSED                                                                      [  9%]
pysm3/tests/test_cmb.py::test_cmb_lensed[c1-100] PASSED                                                                     [ 10%]
pysm3/tests/test_cmb.py::test_cmb_lensed[c1-353] PASSED                                                                     [ 11%]
pysm3/tests/test_co.py::test_co[False] PASSED                                                                               [ 12%]
pysm3/tests/test_co.py::test_co[True] PASSED                                                                                [ 13%]
pysm3/tests/test_co.py::test_co_model[co2] PASSED                                                                           [ 14%]
pysm3/tests/test_co.py::test_co_model[co3] PASSED                                                                           [ 15%]
pysm3/tests/test_dust.py::test_blackbody_ratio PASSED                                                                       [ 16%]
pysm3/tests/test_dust.py::test_dust_model[d0-30] PASSED                                                                     [ 17%]
pysm3/tests/test_dust.py::test_dust_model[d0-100] PASSED                                                                    [ 18%]
pysm3/tests/test_dust.py::test_dust_model[d0-353] PASSED                                                                    [ 19%]
pysm3/tests/test_dust.py::test_dust_model[d1-30] PASSED                                                                     [ 20%]
pysm3/tests/test_dust.py::test_dust_model[d1-100] PASSED                                                                    [ 21%]
pysm3/tests/test_dust.py::test_dust_model[d1-353] PASSED                                                                    [ 22%]
pysm3/tests/test_dust.py::test_dust_model[d2-30] PASSED                                                                     [ 23%]
pysm3/tests/test_dust.py::test_dust_model[d2-100] PASSED                                                                    [ 24%]
pysm3/tests/test_dust.py::test_dust_model[d2-353] PASSED                                                                    [ 25%]
pysm3/tests/test_dust.py::test_dust_model[d3-30] PASSED                                                                     [ 25%]
pysm3/tests/test_dust.py::test_dust_model[d3-100] PASSED                                                                    [ 26%]
pysm3/tests/test_dust.py::test_dust_model[d3-353] PASSED                                                                    [ 27%]
pysm3/tests/test_dust.py::test_dust_model[d6-30] PASSED                                                                     [ 28%]
pysm3/tests/test_dust.py::test_dust_model[d6-100] PASSED                                                                    [ 29%]
pysm3/tests/test_dust.py::test_dust_model[d6-353] PASSED                                                                    [ 30%]
pysm3/tests/test_dust_layers.py::test_modified_black_body_class PASSED                                                      [ 31%]
pysm3/tests/test_dust_layers.py::test_model_d12[100] PASSED                                                                 [ 32%]
pysm3/tests/test_dust_layers.py::test_model_d12[353] PASSED                                                                 [ 33%]
pysm3/tests/test_dust_layers.py::test_model_d12[857] PASSED                                                                 [ 34%]
pysm3/tests/test_dust_pysm3.py::test_dust_model_353[d9] PASSED                                                              [ 35%]
pysm3/tests/test_dust_pysm3.py::test_dust_model_353[d10] PASSED                                                             [ 36%]
pysm3/tests/test_dust_pysm3.py::test_gnilc_857[d9] PASSED                                                                   [ 37%]
pysm3/tests/test_dust_pysm3.py::test_gnilc_857[d10] PASSED                                                                  [ 38%]
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[353] PASSED                                                                 [ 39%]
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[857] PASSED                                                                 [ 40%]
pysm3/tests/test_freefree.py::test_model[f1-30] PASSED                                                                      [ 41%]
pysm3/tests/test_freefree.py::test_model[f1-100] PASSED                                                                     [ 42%]
pysm3/tests/test_freefree.py::test_model[f1-353] PASSED                                                                     [ 43%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d7-100] PASSED                                                           [ 44%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d7-353] PASSED                                                           [ 45%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d7-900] PASSED                                                           [ 46%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d5-100] PASSED                                                           [ 47%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d5-353] PASSED                                                           [ 48%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d5-900] PASSED                                                           [ 49%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d8-100] PASSED                                                           [ 50%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d8-353] PASSED                                                           [ 50%]
pysm3/tests/test_hd17.py::test_highfreq_dust_model[d8-900] PASSED                                                           [ 51%]
pysm3/tests/test_interpolating.py::test_interpolating PASSED                                                                [ 52%]
pysm3/tests/test_interpolating.py::test_interpolating_bandpass_boundary_above PASSED                                        [ 53%]
pysm3/tests/test_interpolating.py::test_interpolating_bandpass_boundary_below PASSED                                        [ 54%]
pysm3/tests/test_max_nside.py::test_sky_max_nside PASSED                                                                    [ 55%]
pysm3/tests/test_max_nside.py::test_sky_max_nside_highres PASSED                                                            [ 56%]
pysm3/tests/test_read_map.py::test_read_map_unit PASSED                                                                     [ 57%]
pysm3/tests/test_read_map.py::test_read_map_unit_dimensionless PASSED                                                       [ 58%]
pysm3/tests/test_smoothing.py::test_smoothing_healpix PASSED                                                                [ 59%]
pysm3/tests/test_smoothing.py::test_car_nosmoothing PASSED                                                                  [ 60%]
pysm3/tests/test_smoothing.py::test_healpix_output_nside PASSED                                                             [ 61%]
pysm3/tests/test_synch_pysm3.py::test_synch_model_s7_noscaling[s7] PASSED                                                   [ 62%]
pysm3/tests/test_synch_pysm3.py::test_synch_model_s7_44[s7] PASSED                                                          [ 63%]
pysm3/tests/test_synch_pysm3.py::test_synch_model_noscaling[s4] PASSED                                                      [ 64%]
pysm3/tests/test_synch_pysm3.py::test_synch_model_noscaling[s5] PASSED                                                      [ 65%]
pysm3/tests/test_synch_pysm3.py::test_synch_44[s4] PASSED                                                                   [ 66%]
pysm3/tests/test_synch_pysm3.py::test_synch_44[s5] PASSED                                                                   [ 67%]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[23] PASSED                                                                   [ 68%]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[44] PASSED                                                                   [ 69%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s0-30] PASSED                                                       [ 70%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s0-100] PASSED                                                      [ 71%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s0-353] PASSED                                                      [ 72%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s1-30] PASSED                                                       [ 73%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s1-100] PASSED                                                      [ 74%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s1-353] PASSED                                                      [ 75%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s2-30] PASSED                                                       [ 75%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s2-100] PASSED                                                      [ 76%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s2-353] PASSED                                                      [ 77%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s3-30] PASSED                                                       [ 78%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s3-100] PASSED                                                      [ 79%]
pysm3/tests/test_synchrotron.py::test_synchrotron_model[s3-353] PASSED                                                      [ 80%]
pysm3/tests/test_unit_conversion_pysm2.py::TestConvertUnits::test_convert_units PASSED                                      [ 81%]
pysm3/tests/test_units.py::test_conversion PASSED                                                                           [ 82%]
pysm3/tests/test_units.py::test_fits_unit_functionality PASSED                                                              [ 83%]
pysm3/tests/test_utils.py::test_get_relevant_frequencies PASSED                                                             [ 84%]
pysm3/tests/test_utils.py::test_has_polarization PASSED                                                                     [ 85%]
pysm3/tests/test_utils.py::test_bandpass_unit_conversion PASSED                                                             [ 86%]
pysm3/tests/test_utils.py::test_bandpass_integration_tophat PASSED                                                          [ 87%]
pysm3/tests/test_utils.py::test_trapz[uniform] PASSED                                                                       [ 88%]
pysm3/tests/test_utils.py::test_trapz[non-uniform] PASSED                                                                   [ 89%]
pysm3/tests/test_utils.py::test_bandpass_integration_weights PASSED                                                         [ 90%]
pysm3/tests/test_utils.py::test_remotedata PASSED                                                                           [ 91%]
pysm3/tests/test_utils.py::test_remotedata_globalpath PASSED                                                                [ 92%]
pysm3/tests/test_utils.py::test_add_metadata PASSED                                                                         [ 93%]
pysm3/tests/test_utils.py::test_add_metadata_different_units PASSED                                                         [ 94%]
pysm3/tests/test_utils.py::test_data_raise PASSED                                                                           [ 95%]
pysm3/tests/test_websky.py::test_SPT_CIB_map_scaling PASSED                                                                 [ 96%]
pysm3/tests/test_websky.py::test_radiogalaxies PASSED                                                                       [ 97%]
pysm3/tests/test_websky.py::test_cib PASSED                                                                                 [ 98%]
pysm3/tests/test_websky.py::test_sz[thermal] PASSED                                                                         [ 99%]
pysm3/tests/test_websky.py::test_sz[kinetic] PASSED                                                                         [100%]

======================================================== warnings summary =========================================================
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[353]
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[857]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[23]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[44]
  /global/common/software/cmb/zonca/conda/pycmb/lib/python3.10/site-packages/healpy/fitsfunc.py:643: ComplexWarning: Casting complex values to real discards the imaginary part
    alm.real[i] = almr

pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[353]
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[857]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[23]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[44]
  /global/common/software/cmb/zonca/conda/pycmb/lib/python3.10/site-packages/healpy/fitsfunc.py:644: ComplexWarning: Casting complex values to real discards the imaginary part
    alm.imag[i] = almi

pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[353]
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[857]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[23]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[44]
  /global/common/software/cmb/zonca/conda/pycmb/lib/python3.10/site-packages/healpy/sphtfunc.py:607: ComplexWarning: Casting complex values to real discards the imaginary part
    (np.asarray(cl, dtype=np.float64) if cl is not None else None)

pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[353]
pysm3/tests/test_dust_pysm3.py::test_d10_vs_d11[857]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[23]
pysm3/tests/test_synch_pysm3.py::test_s6_vs_s5[44]
  /global/common/software/cmb/zonca/conda/pycmb/lib/python3.10/site-packages/healpy/sphtfunc.py:563: ComplexWarning: Casting complex values to real discards the imaginary part
    cls_list = [np.asarray(cls, dtype=np.float64)]

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================== 104 passed, 2 skipped, 16 warnings in 990.11s (0:16:30) =====================================

zonca commented 1 year ago

as discussed on the PanExperiment Galascic Science group call, the implementation of the new emission models in PySM 3.4.0b9 looks stable and has been extensively used in simulations by different experiments, for example:

The plan is to fix issue #155 about renaming templates to avoid caching issues for people that had previously installed a beta release and release 3.4.0.

Please write here if you have any other feedback about the release.

zonca commented 11 months ago

3.4.0 released now, see changelog for more details.

galsci / pysm