Open Chanuntorn opened 2 years ago
@Chanuntorn the following procedure worked for me just now, at least at NERSC:
Log into NERSC. Following the instructions here, create a dedicated (local) conda environment and install Revolver
's dependencies (plus a couple more handy libraries like matplotlib
):
module load python
conda create -n desivoids python=3.8
conda activate desivoids
conda install -y numpy scipy astropy cython ipython matplotlib
pip install healpy pyfftw
A few notes regarding the preceding step:
desivoids
but you can call it whatever you want.pyfftw
doesn't work with python v3.9
, so you have to use v3.8
at least for now.conda activate desivoids
into your terminal without doing module load python
. (In detail, conda
modifies your .bashrc
to run conda init
every time you log in, which is not great practice, but here we are.)Next, create a working directory (obviously choose your own path) and then clone the Revolver
code base into that directory. Now, before running make
I had to hand-edit line 14 of the Makefile
to change make -C src all
to make -C src all_nompi
, as instructed in the README
if MPI wasn't available. I did try module load openmpi
as instructed here first, but Revolver
couldn't find the appropriate library and header files. It would be great to try to figure this out in future. In any case, do:
mkdir /global/cscratch1/sd/ioannis/voids
cd /global/cscratch1/sd/ioannis/voids
git clone https://github.com/seshnadathur/Revolver.git
cd Revolver
[edit Makefile]
make
Next, to reproduce the SDSS/CMASS results, download the large-scale structure galaxy and random catalogs:
cd /global/cscratch1/sd/ioannis/voids
wget https://data.sdss.org/sas/dr12/boss/lss/galaxy_DR12v5_CMASS_South.fits.gz
wget https://data.sdss.org/sas/dr12/boss/lss/random0_DR12v5_CMASS_South.fits.gz
gunzip *.fits.gz
Next, you need to modify the /global/cscratch1/sd/ioannis/voids/Revolver/parameters/params.py
file to point to these catalogs. Use a text editor to update the paths and might as well use 32 threads. Note that the trailing slash on the output_folder
directory name is crucial! Here's what my params.py
file looks like
handle = 'DR12_CMASS_South'
output_folder = '/global/cscratch1/sd/ioannis/voids/out/'
tracer_file = '/global/cscratch1/sd/ioannis/voids/galaxy_DR12v5_CMASS_South.fits'
random_file = '/global/cscratch1/sd/ioannis/voids/random0_DR12v5_CMASS_South.fits'
mask_file = 'masks/DR12v5_CMASS_South_mask.fits'
void_prefix = 'recon-voids'
verbose = True
do_recon = True
use_barycentres = False
run_voxelvoids = True
do_tessellation = True
nthreads = 32
Finally you can run Revolver
, which for me took about 4 minutes on a login node (with 32 threads)!
cd /global/cscratch1/sd/ioannis/voids/Revolver
time python revolver.py --par parameters/params.py
Loading parameters from parameters/params.py
==== Running reconstruction for real-space positions ====
Loading galaxy data from file...
Loading randoms data from file...
==== Starting the reconstruction ====
Using values of growth rate f = 0.780 and bias b = 2.000
Smoothing scale [Mpc/h]: 10.0
Number of bins: 512
Box size [Mpc/h]: 2904.55
Bin size [Mpc/h]: 5.67
Loop 0
Reading wisdom from wisdom.512.32
Creating FFTW objects...
Allocating randoms in cells...
Smoothing...
Allocating galaxies in cells...
Smoothing galaxy density field ...
Computing density fluctuations, delta...
Fourier transforming delta field...
Inverse Fourier transforming to get psi...
Calculating shifts...
Loop 1
Allocating galaxies in cells...
Smoothing galaxy density field ...
Computing density fluctuations, delta...
Fourier transforming delta field...
Inverse Fourier transforming to get psi...
Calculating shifts...
Loop 2
Allocating galaxies in cells...
Smoothing galaxy density field ...
Computing density fluctuations, delta...
Fourier transforming delta field...
Inverse Fourier transforming to get psi...
Calculating shifts...
==== Done reconstruction ====
Reconstruction took 44.655 seconds Loading galaxy data from file...
==== Starting the void-finding with voxel-based method ==== 217780 tracers found Box size [Mpc/h]: 2544.546 Initial bin size [Mpc/h]: 12.79, nbins = 199 Final bin size [Mpc/h]: 5.68, nbins = 448 Smoothing scale [Mpc/h]: 18.29 Allocating galaxies in cells... Allocating randoms in cells... Smoothing density fields ... Finding density minima Total number of voxels: 89915392 Reading in the dens data from file ... Setting the voxel adjacencies ... Finding jumper for each voxel About to jump ... Post-jump ... 10979 zones found... Finding zone borders Allocating zone adjacencies and links Finding weakest links Found zone adjacencies delta ranges from -9.945806e-01 to 4.009290e+00. Writing the zone memberships Post-processing voids Total 2613 voids pass all cuts ==== Finished with voxel-based method ==== Voxel voids took 39.823 seconds
==== Starting the void-finding with ZOBOV ==== Loading galaxy data from file... Cutting galaxies outside the redshift limits provided WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version. [python_tools.zobov] Kept 208491 tracers after all cuts Determining survey redshift selection function ... Generating buffer mocks around survey edges ... buffer mocks will have 10.0 x the galaxy number density WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version. [python_tools.zobov] placed 135214 buffer mocks at high-redshift cap placed 55543 buffer mocks at low-redshift cap WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version. [python_tools.zobov] placed 354366 buffer mocks along the survey boundary edges Using box length 2774.22 added 24292 guards to stabilize the tessellation Buffer mocks written to file /global/cscratch1/sd/ioannis/voids/out/DR12_CMASS_South_mocks.npy Calling vozisol to do the tessellation... Tessellation done. Post-processing voids ... Identified 2138 potential voids. Now extracting circumcentres ... WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version. [python_tools.zobov] Removed 0 edge-contaminated voids ==== Finished with ZOBOV-based method ==== ZOBOV took 132.518 seconds
real 4m8.975s user 6m8.724s sys 0m58.498s
For the record I did try to do all of this in a Docker/shifter container (including MPI), but the way the Revolver
package is set up made it challenging to add the Python code to PYTHONPATH
and to also use a parameter file in a non-standard location. But that's a different ticket!
Thanks for looking into this and suggesting the workaround – I have not had time to test Revolver out on NERSC yet. A few questions/comments:
module load openmpi
and things have worked fine.voxel
void-finding) can require a lot of memory because of needing an N^3 grid. On some HPC systems it might be necessary to increase the number of threads to get sufficient memory. This is an efficiency issue for Revolver that needs work.I should also mention that as soon as I have the time, I am planning a major rewrite of all the Revolver code, including transitioning to using pyrecon
to provide the reconstruction, and using YAML input in place of the current system. This will change some of the workflow, but hopefully in a positive way, and making it more consistent with other DESI code.
An error occurs when running
python revolver.py --par parameters/params.py
.Steps I completed after logging into jupyter.nersc
Then I edited parameters/params.py to match my file paths after downloading the SDSS files. In the Makefile, I changed the line
make -C src all
tomake -C src all_nompi
. I proceed with the following