search

alessiospuriomancini / cosmopower

Machine Learning - accelerated Bayesian inference
https://alessiospuriomancini.github.io/cosmopower
GNU General Public License v3.0
58 stars 26 forks source link

More documentation of accuracy #11

Closed itrharrison closed 2 years ago

itrharrison commented 2 years ago

When implementing CosmoPower for SOLiket (see here) we evaluated the accuracy against CAMB and CLASS runs, finding O(1.e-2) accuracy.

It would be great if this expectation could have been compared to something documented here, in order to understand whether out implementation had succeeded.

I am aware of the accuracy plots in the paper, but because they require the sigma_ell corresponding to SO they are not as intuitive or easy to generate.

alessiospuriomancini commented 2 years ago

hi @itrharrison, we also have accuracy plots without reference to the SO sigma_ell. @dpiras will post one of them below. We would need more details about your CosmoPower model to evaluate the accuracy you quoted -- did you re-train or did you use some pre-trained model? Anyway, the sigma_ell curves are actually from the SO repository, https://github.com/simonsobs/so_noise_models . For details, you can look at the notebooks in the CosmoPower repository -- there we download these curves in the final part, where we test the accuracy of the trained emulator on the testing spectra.

dpiras commented 2 years ago

These are the accuracy plots, before dividing by sigma_ell given by SO, for TT; they are typically within O(1e-3). I attach 3 samples here, hope you can read the labels. Happy to answer any other questions! image

itrharrison commented 2 years ago

Brilliant, thank you! We are using the pre-trained networks:

    network_path: str = "CP_paper/CMB"
    cmb_tt_nn_filename: str = "cmb_TT_NN"
    cmb_te_pcaplusnn_filename: str = "cmb_TE_PCAplusNN"
    cmb_ee_nn_filename: str = "cmb_EE_NN"

and when comparing to a CLASS-generated spectra we also got O(1.e-3) agreement 👍 . I'm happy with being able to find those plots again in this Issue, but I guess they could also be added in the getting_started notebooks?

alessiospuriomancini commented 2 years ago

thanks for the suggestion @itrharrison, will do!