Mismatch between posteriors from polychord vs importance sampler

[jessmuir]

Hello! I've been investigating something that was initially flagged by Noah (@nweaverd), where the posterior reported by the importance sampler and polychord sampler slightly differ in some cases even when they are run with the exact same cosmosis pipeline.

The context is that occasionally we've been running an importance sampler on chains as an approach to save model predictions for datavectors that can be used for making plots, PPD calculations, and some validation tests. In principle, doing this with the exact same cosmosis pipeline, same data, same scale cuts, should produce importance sampler output where all the log_weight entries are 0, meaning the IS post is identical to the original sampler's old_post. We're finding that this isn't always the case.

(There's a thread on this in the cosmosis channel of the DES slack team; let me know if you want to be pointed to that.)

We're still not sure what's happening, but compiling notes on what we've checked so far:

• The list and importance samplers produce matching posteriors.
• Including or not including extra_output columns in either the original or IS chain doesn't affect this.
• Noah did some extensive investigation looking at whether the IS log_weight correlates with anything; it doesn't have any obvious correlation with post, like, prior, old_post, or the handful of individual parameters we checked.
• Looking back at DES Y3 chains where we used this, and re-checking with some newer simulated Y6 chains we do see differences in the polychord and IS posts. Generally more constraining chains show less differences. E.g. Planck+BAO+SN has smaller missmatches than Planck alone, DES+external is smaller than DES alone. However, 3x2pt has bigger differences than shear only.
• I don't see this mismatch when I ran polychord and then importance sampler for the cosmosis example2 setup for the pantheon SNe likelihood.
• I don't see this mismatch when I ran a simplified DES chain sampling only A_s and Omega_m.
• I do see the mismatch when I ran a simplified DES chain sampling only one of the shear calibration parameters. This makes me think that this may have something to do with how priors are evaluated, specifically parameters with Gaussian priors. This might explain why 3x2pt shows bigger differences than shear only, since there are more nuisance parameters with non-flat priors.
• However, when I try adding a Gaussian prior to parameters for the pantheon SNe chain, it doesn't affect agreement between polychord and IS. So if this is related to Gaussian priors, it's not only that.

We have a couple more ideas of things to check. I'll add notes here as we get to them.

I know that running a test sampler handles priors differently that running a sampler with varying parameters, so it seems plausible that there might be worth scrutinizing how the importance and list samplers do this, but I don't have a good enough understanding to be sure.

[joezuntz]

I haven't been able to replicate this directly. Could you confirm from looking at the chain headers that the exact same cosmosis-standard-library version was used in each case? And perhaps give me some more detail on exactly what you ran to replicate?

[jessmuir]

Hm. I just attempted to reproduce the vary-m1-only result on my laptop, and am finding matching polychord and IS posteriors.

Since I'd previously ran the test on my office computer, I won't have access to those output files for another week or so (I'm traveling). I'm relatively sure that in that case I was using the same CSL version since I ran polychord then IS in succession with no code updates between them.

Previously I'd seen mismatches when using IS to save extra derived parameters for DES Y3 chains. I'll aim to hunt down the details on those within the next few days.

[nweaverd]

Sorry for the delayed response to this! I was actually running with Nautilus rather than Polychord (an on NERSC). I was worried about different versions so ran a new Nautilus chain and then ran the Importance Sampler using the exact same params.ini, just changing the sampler, with only a few hours in between and no code updates.

Here's a portion of the Nautilus output header:

#cosmological_parameters--omega_m   cosmological_parameters--h0 cosmological_parameters--omega_b    cosmological_parameters--n_s    cosmological_parameters--a_s    cosmological_parameters--mnu    shear_calibration_parameters--m1    shear_calibration_parameters--m2    shear_calibration_parameters--m3    shear_calibration_parameters--m4    wl_photoz_errors--bias_1    wl_photoz_errors--bias_2    wl_photoz_errors--bias_3    wl_photoz_errors--bias_4    lens_photoz_errors--bias_1  lens_photoz_errors--bias_2  lens_photoz_errors--bias_3  lens_photoz_errors--bias_4  lens_photoz_errors--width_1 lens_photoz_errors--width_2 lens_photoz_errors--width_3 lens_photoz_errors--width_4 bias_lens--b1   bias_lens--b2   bias_lens--b3   bias_lens--b4   intrinsic_alignment_parameters--a1  intrinsic_alignment_parameters--a2  intrinsic_alignment_parameters--alpha1  intrinsic_alignment_parameters--alpha2  intrinsic_alignment_parameters--bias_ta COSMOLOGICAL_PARAMETERS--SIGMA_8    COSMOLOGICAL_PARAMETERS--SIGMA_12   DATA_VECTOR--2PT_CHI2   log_weight  prior   post
#sampler=nautilus
#n_varied=31
#module_0=consistency 1.1
#module_1=BBN-Consistency 0705.0290
## CITE Comp.Phys.Commun.178:956-971,2008
#module_2=camb 2.0
## CITE http://arxiv.org/abs/1201.3654
## CITE http://arxiv.org/abs/astro-ph/9911177
#module_3=fast_pt 1
## CITE https://iopscience.iop.org/article/10.1088/1475-7516/2016/09/015
#module_4=load_nz_fits 1
#module_5=photoz_bias 1
#module_6=photoz_bias 1
#module_7=photoz_bias 1
#module_8=IA #name from ini file
#module_9=project_2d 1.0
#module_10=project_2d 1.0
#module_11=add_magnification 1
#module_12=add_intrinsic 1.0
#module_13=2pt_shear #name from ini file
#module_14=2pt_gal #name from ini file
#module_15=2pt_gal_shear #name from ini file
#module_16=shear_bias 1
#module_17=add_point_mass #name from ini file
#module_18=2pt_like #name from ini file
#module_19=shear_ratio_like #name from ini file
#timestamp=2024-02-16T04:25:34.409664
#platform=Linux-5.14.21-150400.24.81_12.0.87-cray_shasta_c-x86_64-with-glibc2.31
#platform_version=#1 SMP Thu Jan 11 20:03:21 UTC 2024 (e30c7c1)
#uuid=3d638c3adfce4848b081c7d452b55670
#cosmosis_git_version=
#csl_git_version=
#cwd_git_version=6cd5ddd11b7ef6219eff042ecef8a85dbd2c6510
#hostname=nid004207
#username=nweaverd
#workdir=/global/cfs/cdirs/des/nweaverd/cosmosis-standard-library
#n_live=1024
#n_update=1024
#enlarge_per_dim=1.1
#n_points_min=81
#split_threshold=100.0
#n_networks=4
#n_batch=256
#seed=-1
#resume=True
#f_live=0.01
#n_shell=512
#n_eff=10000.0
#discard_exploration=False
#verbose=True
## START_OF_PARAMS_INI
## [DEFAULT]
## 2pt_file = likelihood/des-y3/2pt_NG_final_2ptunblind_02_26_21_wnz_maglim_covupdate.fits
## 
## [runtime]
## sampler = nautilus
## verbosity = standard
## resume = T
## 
## [pipeline]
## modules = consistency  bbn_consistency   camb  fast_pt   fits_nz  lens_photoz_width  lens_photoz_bias  source_photoz_bias    IA  pk_to_cl_gg  pk_to_cl   add_magnification  add_intrinsic    2pt_shear 2pt_gal  2pt_gal_shear    shear_m_bias   add_point_mass   2pt_like  shear_ratio_like
## timing = F
## debug = F
## priors = examples/des-y3-maglim-priors.ini
## values = examples/des-y3-maglim-values.ini
## extra_output = cosmological_parameters/sigma_8 cosmological_parameters/sigma_12 data_vector/2pt_chi2
## fast_slow = F
## first_fast_module = shear_m_bias

And here is the corresponding IS one:

#cosmological_parameters--omega_m   cosmological_parameters--h0 cosmological_parameters--omega_b    cosmological_parameters--n_s    cosmological_parameters--a_s    cosmological_parameters--mnu    shear_calibration_parameters--m1    shear_calibration_parameters--m2    shear_calibration_parameters--m3    shear_calibration_parameters--m4    wl_photoz_errors--bias_1    wl_photoz_errors--bias_2    wl_photoz_errors--bias_3    wl_photoz_errors--bias_4    lens_photoz_errors--bias_1  lens_photoz_errors--bias_2  lens_photoz_errors--bias_3  lens_photoz_errors--bias_4  lens_photoz_errors--width_1 lens_photoz_errors--width_2 lens_photoz_errors--width_3 lens_photoz_errors--width_4 bias_lens--b1   bias_lens--b2   bias_lens--b3   bias_lens--b4   intrinsic_alignment_parameters--a1  intrinsic_alignment_parameters--a2  intrinsic_alignment_parameters--alpha1  intrinsic_alignment_parameters--alpha2  intrinsic_alignment_parameters--bias_ta COSMOLOGICAL_PARAMETERS--SIGMA_8    COSMOLOGICAL_PARAMETERS--SIGMA_12   DATA_VECTOR--2PT_CHI2   cosmological_parameters--sigma_8    cosmological_parameters--sigma_12   data_vector--2pt_chi2   log_weight  prior   old_post    log_weight  post
#sampler=importance
#n_varied=31
#module_0=consistency 1.1
#module_1=BBN-Consistency 0705.0290
## CITE Comp.Phys.Commun.178:956-971,2008
#module_2=camb 2.0
## CITE http://arxiv.org/abs/1201.3654
## CITE http://arxiv.org/abs/astro-ph/9911177
#module_3=fast_pt 1
## CITE https://iopscience.iop.org/article/10.1088/1475-7516/2016/09/015
#module_4=load_nz_fits 1
#module_5=photoz_bias 1
#module_6=photoz_bias 1
#module_7=photoz_bias 1
#module_8=IA #name from ini file
#module_9=project_2d 1.0
#module_10=project_2d 1.0
#module_11=add_magnification 1
#module_12=add_intrinsic 1.0
#module_13=2pt_shear #name from ini file
#module_14=2pt_gal #name from ini file
#module_15=2pt_gal_shear #name from ini file
#module_16=shear_bias 1
#module_17=add_point_mass #name from ini file
#module_18=2pt_like #name from ini file
#module_19=shear_ratio_like #name from ini file
#timestamp=2024-02-16T12:47:53.710098
#platform=Linux-5.14.21-150400.24.81_12.0.87-cray_shasta_c-x86_64-with-glibc2.31
#platform_version=#1 SMP Thu Jan 11 20:03:21 UTC 2024 (e30c7c1)
#uuid=64f5b4cc3aee40a88d86f80fa2c8f77c
#cosmosis_git_version=
#csl_git_version=
#cwd_git_version=6cd5ddd11b7ef6219eff042ecef8a85dbd2c6510
#hostname=nid005704
#username=nweaverd
#workdir=/global/cfs/cdirs/des/nweaverd/cosmosis-standard-library
#input=output/des-y3-maglim_naut_v2.txt
#nstep=128
#add_to_likelihood=False
## START_OF_PARAMS_INI
## [DEFAULT]
## 2pt_file = likelihood/des-y3/2pt_NG_final_2ptunblind_02_26_21_wnz_maglim_covupdate.fits
## 
## [runtime]
## sampler = importance
## verbosity = standard
## 
## [pipeline]
## modules = consistency  bbn_consistency   camb  fast_pt   fits_nz  lens_photoz_width  lens_photoz_bias  source_photoz_bias    IA  pk_to_cl_gg  pk_to_cl   add_magnification  add_intrinsic    2pt_shear 2pt_gal  2pt_gal_shear    shear_m_bias   add_point_mass   2pt_like  shear_ratio_like
## timing = F
## debug = F
## priors = examples/des-y3-maglim-priors.ini
## values = examples/des-y3-maglim-values.ini
## extra_output = cosmological_parameters/sigma_8 cosmological_parameters/sigma_12 data_vector/2pt_chi2
## fast_slow = F
## first_fast_module = shear_m_bias

Interestingly the cosmosis and CSL git versions are blank, but like I said they should be identical. The old_post column of the IS matches the Nautilus chain, as do all the parameter values, but the post column differs with a scatter of ~1.

[joezuntz]

Could you send me the full chains, or just the full ini files that generated them?

[jessmuir]

Just to follow up: I'm now unable to replicate the mismatch I noted in my post above where I said

I do see the mismatch when I ran a simplified DES chain sampling only one of the shear calibration parameters. This makes me think that this may have something to do with how priors are evaluated, specifically parameters with Gaussian priors. This might explain why 3x2pt shows bigger differences than shear only, since there are more nuisance parameters with non-flat priors.

I don't know if I had an error in my initial test, or some update to my workspace or to cosmosis in the meantime affected this. In any case, now on my simple desktop tests I'm unable to replicate the polychord-vs-IS mismatch. Apologies for adding confusion to the discussion!