simonsobs / BBPipe

B-modes pipeline constructor
5 stars 6 forks source link

Update on bandpass and polarization angle marginalization #23

Closed mabitbol closed 4 years ago

mabitbol commented 5 years ago

Here we begin to address bandpass and polarization angle systematics. I will break the results into two categories (1) addressing biases from an incomplete model and (2) assessing the increase in sigma r from marginalizing over systematics.

1) Check for biases from incomplete models. (If we cannot include in the model or have a poor model then we would like to assess the potential for bias) (i) Impact of $\phi(\nu)$: the in-band frequency dependent polarization response/rotation. $\phi(\nu)$ can be generated from the broadband 3-layer AHWP or the sinuous antenna design. (ii) Other? Intrinsic foreground EB? (iii) Can we create any intuition or rules of thumb to help understand what shapes and levels of systematics are most critical?

TODO list

2) Assess impact on ability to constrain r. How much does our constraint on r degrade when marginalizing over additional foreground and systematic parameters. We are considering bandpass and polarization angle systematics here in particular. (i) Bandpass uncertainties. Gain, shift to parameterize uncertainty. +Jeff's model of slopes, shifts, ripples. (ii) Polarization angle uncertainties. Including $\phi(\nu)$ uncertainties. Intrinsic EB.

TODO list

General TODO list

Other issues:

Another important note: These results do not agree with the SO forecast paper because of the simplified foreground model. There is no residual FG marginalization here that drove up the errors in the forecast! Things might be more realistic and agree with the forecast when we go to data model 3.

mabitbol commented 5 years ago

Note: You can click on the figures for a bigger version. I have higher resolution ones if you want them (github has 10MB limit). Please let me know if you use any figures.

Results so far 2019-10-08 Data model 1

Other things:

Baseline constraints as printed by get_dist (can anyone help with the formatting here? Github inline latex or a different get_dist print function?)

\begin{tabular} { l c}

Parameter & 95\% limits\ \hline {\boldmath$A{lens} $} & $0.991^{+0.056}{-0.055} $\

{\boldmath$r $} & $< 0.00379 $\

{\boldmath$\betad $} & $1.591^{+0.032}{-0.032} $\

{\boldmath$\epsilon $} & $0.000^{+0.040}_{-0.040} $\

{\boldmath$\alpha^{EE}d $} & $-0.422^{+0.077}{-0.077} $\

{\boldmath$A^{EE}d $} & $10.03^{+0.71}{-0.66} $\

{\boldmath$\alpha^{BB}d $} & $-0.202^{+0.079}{-0.080} $\

{\boldmath$A^{BB}d $} & $5.01^{+0.37}{-0.35} $\

{\boldmath$\betas $} & $-3.005^{+0.065}{-0.067} $\

{\boldmath$\alpha^{EE}s $} & $-0.61^{+0.25}{-0.26} $\

{\boldmath$A^{EE}s $} & $4.02^{+0.43}{-0.40} $\

{\boldmath$\alpha^{BB}s $} & $-0.41^{+0.28}{-0.30} $\

{\boldmath$A^{BB}s $} & $2.01^{+0.26}{-0.24} $\ \hline \end{tabular}

Baseline (r pdf): bbpipe_baseline_r

Bandpass marginalized (r pdf): bbpipe_bands_r_pdf_norm

Angles marginalized (r pdf): bbpipe_angles0_r_pdf_norm

$\phi(\nu)$ in data but not model (r pdf): bbpipe_sinuous_ahwp_r_pdf2_norm

Baseline full triangle plot: bbpipe_baseline_fulltriangle

All of the above triangle plot (CMB + FG params shown): bbpipe_baseline_bands_angles0_phinus_cmbfg_triangle_norm

mabitbol commented 5 years ago

Update November 7th 2019. Updated foreground model with params drawn on 10-20% of the sky from Planck, SPASS etc. Used wider priors and made r two sided.

\begin{tabular} { l c}

Parameter & 95\% limits\ \hline {\boldmath$A{lens} $} & $1.000^{+0.064}{-0.062} $\

{\boldmath$r $} & $0.0002^{+0.0037}_{-0.0033}$\

{\boldmath$\betad $} & $1.530^{+0.015}{-0.015} $\

{\boldmath$\epsilon $} & $0.200^{+0.036}_{-0.036} $\

{\boldmath$\alpha^{EE}d $} & $-0.302^{+0.068}{-0.069} $\

{\boldmath$A^{EE}d $} & $40.1^{+2.7}{-2.5} $\

{\boldmath$\alpha^{BB}d $} & $-0.151^{+0.069}{-0.069} $\

{\boldmath$A^{BB}d $} & $20.0^{+1.3}{-1.3} $\

{\boldmath$\betas $} & $-3.201^{+0.045}{-0.046} $\

{\boldmath$\alpha^{EE}s $} & $-1.31^{+0.23}{-0.24} $\

{\boldmath$A^{EE}s $} & $8.03^{+0.76}{-0.70} $\

{\boldmath$\alpha^{BB}s $} & $-1.21^{+0.26}{-0.27} $\

{\boldmath$A^{BB}s $} & $4.02^{+0.44}{-0.41} $\ \hline \end{tabular}

r pdf bbpipe_baseline_r_pdf_norm_mock1

mabitbol commented 4 years ago

28/11/2019 update We have moved to a more aggressive foreground model that I am calling v3 with the following parameters: A_sync_BB = 5. EB_sync = 2. (this is just the ratio of EE to BB not EB) alpha_sync_EE = -0.8 alpha_sync_BB = -0.6 beta_sync = -3.1 nu0_sync = 23.

A_dust_BB = 20. EB_dust = 2. alpha_dust_EE = -0.4 alpha_dust_BB = -0.2 beta_dust = 1.53 temp_dust = 19.6 (fixed in all fits) nu0_dust = 353.

epsilon = 0.2 (synch - dust correlation) fg_eb = 0. if do_eb: fg_eb = 0.1 (this is the intrinsic EB fraction) Alens = 1. (no delensing)

I ran the following MCMCs: baseline, baseline + AHWP, baseline + sinuous, angles, shifts, gains + shifts, off center gains, combined gains angles, dphi1.

TODO

Results None of the above runs produce any bias on r or increase on sigma r. Still trying.......

BB signals: foregrounds, noise, r=0.01 tensors (no lensing) foregrounds_noise_r01_3d

I have a collection of all the plots here: SO_plots

In particular you can check out the r pdfs, the triangle plots, and the constraints.txt's. The plots of the walkers and chains are for diagnostics.

Things to keep in mind

mabitbol commented 4 years ago

Update 05/12/2019 Here is an update on the estimates of biases from the gain, shift, and angle systematics.

Method

I use the FGv3 model to make our dataset and then nearly the exact same model to with a maximum likelihood optimizer. The difference is that in the fitting model I include a fixed systematic effect. For example, if I want to test the impact of mis-modeling the LF1 gain by 5%, I would fix the value of the gain there to 1.05 and then perform the fit with this model. This will of course induce a bias in the params, in particular the FG SED spectral indices which would then presumably leak into r at some level. The dimension of the space is too large to explore with a grid (see note) search so instead I break this down into 3 cases: vary a given systematic on a per channel basis one at a time, vary the systematic for either the LF, MF, or HF bands pairwise in the same direction (e.g. both LF1 and LF2 get a gain of 1.05), or vary the systematic for LF, MF, or HF pairwise in opposite directions (e.g. LF1 gain=1.05 and LF2 gain = 0.95). I will call these runs "per channel", "symmetric", and "asymmetric" as a shorthand. All of the plots can be found here bias plots.

Note: 6 dimensions so if you only had 10 gain values you wanted to test for each of the frequencies you get a million runs... and then we have 4+ systematics. If anyone has any better ideas that would be great, I tried a weird thing where I fixed r in the MCMC and then sampled over the systematics but that didn't seem to work.

Results

The most interesting thing I find is that the worst case runs are usually the asymmetric ones. In hindsight maybe this is obvious as this produces the biggest difference in the neighboring channels (for gains and shifts at least) which leads to incorrect SED indices which leads to a bias on r. The angle biases can be self-calibrated so no need to worry but in any case the results are >1 deg, 0.2 deg, and 0.4 deg for LF, MF, and HF with r bias < 2e-4. The gains and shifts are more worrisome with all the numbers at the 1% level and less for the HF. HOWEVER, before we panic, we are finding that we can easily marginalize over gain/shift uncertainties with negligible impact on sigma r. I am still trying to understand how this is possible and will post that below. Also I still need to run this with a finer resolution to get more exact numbers, now that we have a better idea of where to zoom in.

Summary of results

Gains Worst case limits for a bias of r < 2e-4: LF gains: ~2% (from asymmetric case) MF gains: ~1% (from asymmetric case) HF gains: <1% (from asymmetric case) -- need to check with finer resolution.

bias_perchannel_gain_zoom_r bias_symmetric_gain_zoom_r bias_asymmetric_gain_zoom_r

Shifts Worst case limits for a bias of r < 2e-4: LF shifts: <1% (from asymmetric case) -- need to check with finer resolution. MF shifts: <1% (from asymmetric case) -- need to check with finer resolution. HF shifts: <1% (from asymmetric case) -- need to check with finer resolution.

bias_perchannel_shift_zoom_r bias_symmetric_shift_zoom_r bias_asymmetric_shift_zoom_r

Angles LF gains: >1 deg (from asymmetric case) -- didn't go wider than 1 deg MF gains: 0.2 deg (from symmetric case) HF gains: 0.4 (from symmetric case) Remember that we can self calibrate better than this so these are not a problem even with FG EB.

bias_perchannel_angle_zoom_r bias_symmetric_angle_zoom_r bias_asymmetric_angle_zoom_r

dphi1

Full table of results

Gains 186 63 63 Per Channel gain r{bias}<1e-3 Channel Min Max 1 0.88 1.15 2 0.85 1.1 3 0.93 1.09 4 0.85 1.08 5 0.96 1.03 6 0.97 1.04 Per Channel gain r{bias}<2e-4 Channel Min Max 1 0.96 1.15 2 0.85 1.03 3 0.97 1.04 4 0.96 1.04 5 0.99 1.01 6 0.99 1.02 Asymmetric Delta gain r{bias}<1e-3 Channel Min Max lf 0.2 -0.1 mf 0.1 -0.08 hf 0.02 -0.04 Asymmetric Delta gain r{bias}<2e-4 Channel Min Max lf 0.18 -0.02 mf 0.04 -0.02 hf 0.0 -0.02 Symmetric gain r{bias}<1e-3 Channel Min Max lf 0.9 1.1 mf 0.9 1.1 hf 0.9 1.1 Symmetric gain r{bias}<2e-4 Channel Min Max lf 0.9 1.1 mf 0.93 1.1 hf 0.9 1.1

Shifts Per Channel shift r{bias}<1e-3 Channel Min Max 1 -0.1 0.04 2 -0.03 0.08 3 -0.05 0.05 4 -0.05 0.03 5 -0.01 0.01 6 -0.01 0.01 Per Channel shift r{bias}<2e-4 Channel Min Max 1 -0.06 0.01 2 -0.01 0.05 3 -0.02 0.02 4 -0.02 0.01 5 0.0 0.0 6 0.0 0.0 Asymmetric Delta shift r{bias}<1e-3 Channel Min Max lf 0.02 -0.08 mf 0.04 -0.04 hf 0.0 0.0 Asymmetric Delta shift r{bias}<2e-4 Channel Min Max lf 0.0 -0.04 mf 0.0 0.0 hf 0.0 0.0 Symmetric shift r{bias}<1e-3 Channel Min Max lf -0.1 0.1 mf -0.1 0.1 hf -0.05 0.06 Symmetric shift r{bias}<2e-4 Channel Min Max lf -0.05 0.1 mf -0.1 0.07 hf -0.02 0.02

Angles Per Channel angle r{bias}<1e-3 Channel Min Max 1 -1.0 1.0 2 -1.0 1.0 3 -0.6 0.6 4 -1.0 1.0 5 -1.0 1.0 6 -1.0 1.0 Per Channel angle r{bias}<2e-4 Channel Min Max 1 -1.0 1.0 2 -1.0 1.0 3 -0.3 0.3 4 -0.8 0.8 5 -1.0 1.0 6 -0.6 0.6 Asymmetric Delta angle r{bias}<1e-3 Channel Min Max lf 2.0 -2.0 mf 2.0 -2.0 hf 2.0 -2.0 Asymmetric Delta angle r{bias}<2e-4 Channel Min Max lf 2.0 -2.0 mf 1.0 -1.0 hf 2.0 -2.0 Symmetric angle r{bias}<1e-3 Channel Min Max lf -1.0 1.0 mf -0.4 0.4 hf -0.9 0.9 Symmetric angle r{bias}<2e-4 Channel Min Max lf -1.0 1.0 mf -0.2 0.2 hf -0.4 0.4

dphi1

TODO

mabitbol commented 4 years ago

Attempting a merge... Polangle lost compatibility with SACC.

damonge commented 4 years ago

I'm gonna close this one (but won't delete the branch), since this has been merged into the BBPower repo https://github.com/simonsobs/BBPower