Validate code against CCL predictions

[damonge]

We should validate the code in a realistic user scenario

• 3D and skewers was already validated in James' paper.
• LSST-like simulation and validate density, shear and magnification for a couple of distant bins.
• Intensity mapping (generate an SKA-like simulation and validate for a couple of frequency bins, both C_ells and Pk^1D - see e.g. https://arxiv.org/abs/1609.00019)

[fjaviersanchez]

More info about the charges:

Charged hours:82,721.96
Raw hours:590.87
Machine hours:20,409.51
CF:1
Secs:32,801
Cpu count:1,024
Alloc nodes:16
Alloc tres:billing=1024,cpu=1024,mem=1888G,node=16

And from the log:

*** Creating Gaussian density field 
Creating Fourier-space density and Newtonian potential 
>    Relative time ellapsed 72210.8 ms
Transforming density and Newtonian potential
>    Relative time ellapsed 233245.1 ms
Normalizing density and Newtonian potential 
>    Relative time ellapsed 2102.5 ms
 <d>=-7.598E-12, <d^2>=6.556E-01

*** Creating physical matter density
Lognormalizin'
>    Relative time ellapsed 41867.3 ms
*** Computing normalization of density field
>    Relative time ellapsed 60795.2 ms

*** Getting point sources
 0-th galaxy population
   Poisson-sampling
   There will be 5896681484 objects in total 
   Assigning coordinates
>    Relative time ellapsed 108992.3 ms

*** Re-distributing sources across nodes
>    Relative time ellapsed 66032.6 ms
*** Getting LOS information
>    Relative time ellapsed 127968042.1 ms

*** Writing kappa source maps
>    Relative time ellapsed 4231531.6 ms

*** Writing source catalogs
 0-th population (FITS)
>    Relative time ellapsed 111753.5 ms

|-------------------------------------------------|

>    Total time ellapsed 132914008.8 ms

[andreufont]

Hi @fjaviersanchez , @damonge - is it possible to change the settings of this folder, such that everyone can access the simulation? This way, both @cramirezpe and @damonge would be able to run their analysis script and compare results.

[fjaviersanchez]

I just changed the permissions so it's public for everyone (I did the same for the previous "fast" version as well). Please let me know if this works. If not, I can try putting the files at some other location.

[cramirezpe]

Nice to have the huge simulation available, I'll see how it compares with the faster shear simulation.

@damonge , you were asking for the d,e1,e2 maps. I've shared the folder where I have all the analysis. In concrete here: /global/cscratch1/sd/cramirez/CoLoRe_CCL/sims/javier/n_4096/ccl_data/20200913_150932/ you can find results for nside=1024. You can load the different outputs using np.loadtxt(filename) (maybe not the most efficient way of storing large data).

---

A part from that, I continued the analysis of what I had until now. I compared the r_smooth=5 simulation with the r_smooth=2.

The first thing to note here is that the r_smooth=5 simulation have n_grid=4095 vs n_grid=4096 for the other one. Is there a reason for that @javier?

Comparing both simulations at the same time we obtain (Sim 5 is the sim with r_smooth=5 and Sim 2 the r_smooth=2 one)

And the ratio sim/prediction (I restricted the y axis to make easy to see low ell regions):

In general the new r_smooth=5 simulation behaves better than the previous r_smooth=2 in low ell regions (as expected) but worse at high ell regions (also as expected).

Then the B-modes:

where we see again that the new r_smooth=5performs better.

---

I also had a comparison between a faster shear and slow shear (for less galaxies than the sim by Javier). But I'm observing non-zero B modes at high ell that I don't see in other sims. This can be an error from my side so I decided to share the plots once I'm confident about them.

[fjaviersanchez]

@cramirezpe there's no reason it's 4095 except that I don't know how to write... 🤦 I can rerun if needed. Apologies!

[fjaviersanchez]

Another option is to make the fast version with n_grid=4095 too :p

[andreufont]

@fjaviersanchez - I wasn't aware you could do a FFT with an odd number of cells. Is it possible that the code is secretely rounding it up to 4096 or 4094?

[fjaviersanchez]

I think that this is not the case anymore. You may lose a bit of performance (I think it might not even be the case with the newer processors) but it will still compute it with an odd number of cells (as far as I know).

[fjaviersanchez]

Anyway, I'm rerunning with 4096 (I got some extra hours at NERSC so let's just use them).