covariances for anisotropic noise pixel variance

[xzackli]

This PR implements covariance estimation methods from the Planck papers, which have an approximate treatment of noise pixel covariance. They are ported from my Julia code, PowerSpectra.jl. This PR currently implements the only the TTTT covariance, using the Python spin-0 interface to generate the eight coupling matrices involved in this calculation. The polarisation covariances are tedious but straightforward, as long as we have gotten TTTT right.

The agreement between the codes is very good, but there is a 0.01% level difference in the basic coupling matrix calculation that I need help understanding.

The basic inputs are

• masks for the four fields in the TTTT (same as existing)
• dict of theory spectra for each of the four fields (same as existing)
• pixel variance maps for the four fields in TTTT (new)
• dict of spectra corresponding to the ratio of the noise power spectrum in each field, divided by the white noise power spectrum associated with the variance map of that field

[codecov-commenter]

Codecov Report

Merging #52 (ab7e2ca) into master (cca8cf1) will increase coverage by 6.49%. The diff coverage is 93.87%.

Additional details and impacted files

[](https://app.codecov.io/gh/simonsobs/pspy/pull/52?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=simonsobs) ```diff @@ Coverage Diff @@ ## master #52 +/- ## ========================================== + Coverage 27.87% 34.36% +6.49% ========================================== Files 18 18 Lines 1980 2028 +48 ========================================== + Hits 552 697 +145 + Misses 1428 1331 -97 ``` | [Impacted Files](https://app.codecov.io/gh/simonsobs/pspy/pull/52?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=simonsobs) | Coverage Δ | | |---|---|---| | [pspy/so\_cov.py](https://app.codecov.io/gh/simonsobs/pspy/pull/52?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=simonsobs#diff-cHNweS9zb19jb3YucHk=) | `25.96% <93.87%> (+18.31%)` | :arrow_up: | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/simonsobs/pspy/pull/52/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=simonsobs)

[xzackli]

Now validated to float32 numerical precision!

[xzackli]

@thibautlouis @zatkins2 this is ready for merge.

[xgarrido]

I'll let @thibautlouis check the code but I wonder if you can put the fits file on a server and download them during the unit test ? There are 3 x 26 Mb files and even if we do not change them, it's quite big. Can we imagine a test with smaller files ?

[xzackli]

It makes sense to automatically download test data, perhaps tagged as part of a pspy GitHub release. I think astropy has an artifacts system?

I chose the test data here to be very low-resolution pixel variance maps with sufficient resolution to have Nyquist frequency at ell ~ 1350, which is the key scale this PR is trying to target.

[xgarrido]

I was thinking to something simple such as adding wget command within the github action script, prior to the test itself.

Of course, it's only 3 x 26 Mb but I don't like the idea that it can cluttered the git history and increase the repository size.

[xzackli]

@xgarrido I've moved the test data to be wget in the CI from a tagged release binary blob on my own fork, and force-pushed my fork to erase the history of the large files (you can check here).

[thibautlouis]

looks good to me, perhaps use @ instead of np.dot here ( analytic_cov = np.dot(np.dot(mbb_inv_ab, analytic_cov), mbb_inv_cd.T)) since Xavier has shown that this is faster , once this is done let's merge this !

[xzackli]

done in https://github.com/simonsobs/pspy/pull/52/commits/1a9ff4c4ed843f3a9ff511d2d5cdb3bfe30eac69