Signal sims 0p5

[ajvanengelen]

This is an update to the script to generate CMB sims, in prep of a run for version 0.5.

This is a partial Pull Request because we have not yet changed the parts that read in sims for the new naming scheme.

Changes include:

• There can now be multiple sets of phi maps (in addition to cmb maps). For instance, if:

  START_CMB_SET: 0
  STOP_CMB_SET: 2
  START_PHI_SET: 0
  STOP_PHI_SET: 2

  then there will be two sets of each of CMB and Phi. Sets 00 and 01 of the CMB will be lensed by each of sets 00 and 01 of Phi.

  • One concern with this is that I changed the interface to util.seed_gen.get_phi_seed(). Will this break reverse compatibility with anything?
  • This means the naming convention for output files has changed to reflect both sets, e.g. fullskyLensedUnabberatedCMB_alm_cmbset00_phiset00_00000.fits

Other tweaks:

• A handy __README.txt file is generated in the sim directory. The idea is to reduce guesswork about what settings went into the set of sims. Here is an example (for a test run of 2 iterations that I did):

  
  This run of /global/u2/e/engelen/actsims/bin/signalSims.py was invoked with config file simsInput_v0p5_test.yaml

Git version number: 9900a91

Parameters were:

PIX_SIZE = 1.0 iStart = 0 STOP_CMB_SET = 2 STOP_PHI_SET = 2 LMAX = 8000 inputSpecRoot = ../data/cosmo2017_10K_acc3 LMAX_NYQ = 10800 doAberration = True START_CMB_SET = 0 iStop = 2 LMAX_WRITE = 5100 dataDir = /global/cscratch1/sd/engelen/simsS1516_v0.5/data/ START_PHI_SET = 0

The run took 7127.668854 seconds.

The run was done on 1 MPI processes.

* The config file, which used to be imported with `flipperDict`, is now written in `yaml`.
* Renaming of CMB files such that "Unabberated" or "Abberated" are always explicitly specified
 * See the example file in `actsims/inputParams/simsInput_v0p5_test.yaml`. Sample results (for 2 iterations) are currently in `/global/cscratch1/sd/engelen/simsS1516_v0.5/data`

[msyriac]

Hi alex, this is looking great. I might dive in and make some edits. In the meantime, could you benchmark a single lensed sim generation run with no aberration? I was finding it takes ~300 seconds, so if it takes 10x long with aberration, we might have to revisit my original suggestion to just re-run everything.

[msyriac]

Here's my benchmark script: https://github.com/ACTCollaboration/actsims/blob/master/bin/siggen.py with some stats in the comments. Note that I used the 'delta_theta' argument to split the lensing up in bands to conserve memory usage. It might not make a lot of difference to the actual time whether you use it or not.

[dwhan89]

• One concern with this is that I changed the interface to util.seed_gen.get_phi_seed(). Will this break reverse compatibility with anything?

I don't think so. If we do, the change will be minimal. Right now, the set_idx for get_phi_seed() is hard-coded to 0/

    def get_phi_seed(self, sim_idx):
        return (0, 0, self.PHI, sim_idx)

This should change to

    def get_phi_seed(self, set_idx, sim_idx):
        return (set_idx, 0, self.PHI, sim_idx)

Hi alex, this is looking great. I might dive in and make some edits. In the meantime, could you benchmark a single lensed sim generation run with no aberration? I was finding it takes ~300 seconds, so if it takes 10x long with aberration, we might have to revisit my original suggestion to just re-run everything.

For a given geometry, the aberration factor needs to be computed only once. So I don't think the aberration should be a bottleneck here. Sigurd has a Aberrator class in pixell (https://github.com/simonsobs/pixell/blob/bf6759ad97d58419e5c7625fe5837e559d34909f/pixell/aberration.py#L54) for this use case.

[ajvanengelen]

Thanks for the quick feedback, Mat and DW!

Mat - sure, I can do that test but probably not for a few days. Go ahead and tweak this script -- the more eyes on it the better.

I think we should strive to make aberrated maps (even if it takes longer) as it could impact several of our science goals. I will look into using the Aberrator class - thanks.

DW, I did do the change you mentioned to get_phi_seed (it should be in the diff for this PR) but I was worried that the calling interface had changed (in case it is called with positional arguments elsewhere).

[msyriac]

Ok, but will we need 4000 aberrated sims? or just a few dozen or few 100?

[ajvanengelen]

I think the baseline should be (once we get the kinks out) that all our sims are always aberrated.

---

From: Mathew S. Madhavacheril notifications@github.com Sent: 02 July 2020 11:30 To: ACTCollaboration/actsims actsims@noreply.github.com Cc: ajvanengelen engelen@cita.utoronto.ca; Author author@noreply.github.com Subject: Re: [ACTCollaboration/actsims] Signal sims 0p5 (#42)

Ok, but will we need 4000 aberrated sims? or just a few dozen or few 100?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ACTCollaboration_actsims_pull_42-23issuecomment-2D653159796&d=DwMCaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=mauVTJTX0aydlVkY5fcx1lQbgv8QnPpRaXAD8fKw_Zs&m=OD_7rtcKjjfB0hVwK0FDIfQvnGZb98osAnFwzxxBtL0&s=BYED8RVokGgP3fU5RhKKie51pBE4EzAWpwT4lPqdlko&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADYBOHBLOS7Q3NVZTBNEBYDRZTG5FANCNFSM4OOFAAMQ&d=DwMCaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=mauVTJTX0aydlVkY5fcx1lQbgv8QnPpRaXAD8fKw_Zs&m=OD_7rtcKjjfB0hVwK0FDIfQvnGZb98osAnFwzxxBtL0&s=4P1hSXEcDtXSj1yG2TelUkO6yQDaWxB6pSvujO9c57E&e=.

[msyriac]

Thanks Alex and DW. New benchmarks: Aberration initialization: ~300 seconds In the loop: Lensing: ~300 seconds Aberration: ~90 seconds

So shouldn't be too much extra time to aberrate all the sims.

[ajvanengelen]

HI Mat, Thanks for doing more tests!!

What machine was this on? For reference, the run for which I posted the __README.txt above was 890 seconds per iteration, doing lensing and aberration (with full initialization each time), on a Cori interactive node with OMP_NUM_THREADS = 32.

The other difference is that I did not explicitly split up the sky into rings like you did.

[msyriac]

My last test didn't do the declination band loop (that only seems to help with memory usage, not speed). I did it on niagara, and used all the 80 hyperthreads. Given that I didn't time the disk i/o, these numbers look pretty much in the same ballpark!

[ajvanengelen]

Great!

[dwhan89]

anged (in case it is called with positional arguments elsewhere).

[dwhan89]

I think things are ready to be merged in.

Alex - I don't think we will have any conflict with get_phi_seed unless someone tries to make v0.4 sims again with old scripts.

[msyriac]

Not yet, I have changes I need to push

[msyriac]

Proposal: let's not save unlensed alms. Those are trivial to generate on-the-fly. Agreed?

[ajvanengelen]

Agreed - I just saved them last time "just in case." We could also not save the phi (or kappa) alm's.

[msyriac]

I think I agree on phi too. If that is the case, we can just increase the ellmax we save the lensed alms out to 10,000. I don't imagine @thibautlouis @stevekchoi will want anything above ell of 10,000? There will however be an effective transfer function for ell>5000 where the lensing becomes inaccurate. We could quantify that and apply it optionally when lensed alms are loaded.

[ajvanengelen]

Sure - in that case though let's make sure to write some helper/wrapper routines (i.e. given just the 'set' and 'iterationNum', generate the alm's or even the maps). This would be similar to what is already in there for generating noise realizations on the fly.

[msyriac]

Made some changes and cleanup. Let me know if ok and we can merge. (No wrapper routines yet) Ideally the same signal sim script would also compare all the spectra to theory and save a transfer function, but also fine to do that separately.

[ajvanengelen]

I just ran a test, looks good! Takes me 2100 seconds for 1 iteration, including 2 CMB sets and 2 Phi sets, on cori with 64 cpus. I'm going to add some loading-up wrappers before proceeding, and will also add the aberration.modulate call