Performance notes of PySM3

galsci / pysm

PySM 3: Sky emission simulations for Cosmic Microwave Background experiments

https://pysm3.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

36 stars 23 forks source link

Performance notes of PySM3 #25

Open zonca opened 5 years ago

zonca commented 5 years ago

Running PySM3 standalone using modifications of this script:

https://github.com/healpy/pysm/blob/master/mpi_examples/pysm3_mpi_so_pysm_models.py

I have all the 8 components requested for the TOD sims, which load a total of 30 maps. Plus bandpass integration of 5 points each for 10 channels and distributed smoothing with libsharp.

zonca commented 5 years ago

It runs fine on Popeye single node with 1 MPI process (48 cores and 768 GB of RAM) at 4096 , using a max of 25GB of RAM in 6 minutes.

zonca commented 5 years ago

Timing, initialize sky and run bandpass integration and beam smoothing for 10 channels:

1 node 1862.6s 23.882 GB
2 nodes 1399.8s 15.213 GB
3 nodes 1169.1s 12.470 GB
5 nodes 978.4s 10.193 GB
10 nodes (4 MPI 12 OMP per node) 702.5s 10.319 GB

Checked for increased memory between channel 0 and 9, seems fine.

zonca commented 5 years ago

Only thing that worries me is that at 10 MPI processes sometimes the MPI job hangs after creating Sky, I guess in one of the all-to-all communications....how do I debug that?? Testing on Cori KNL now.

zonca commented 5 years ago

Breakdown for 10 nodes

10 nodes (4 MPI 12 OMP per node)

Initialization: 111.3s
Create sky and bandpass integration (each channel): 50s
Distributed smoothing (each channel): 6-10s