search

Chris-Pedersen / LaCE_manager

Cosmological emulator for the 1D flux power spectrum of the Lyman-alpha forest
3 stars 1 forks source link

Sampler performance on different clusters #60

Open Chris-Pedersen opened 2 years ago

Chris-Pedersen commented 2 years ago

Plotting a few runs of the small_test.config file (NB you will have to reduce the number of free IGM parameters in the multiprocess_sampler.py script when running this) small_test

We see our lovely islands as usual, but the results are consistent. Will add a comparison of timings a bit later on so we can keep track of performance.

Chris-Pedersen commented 2 years ago

Performance is significantly better at Flatiron - on a 40 core node the test runs for 3300 steps in 240s, vs on the cores24 nodes on Hypatia where we have 5200 steps over 1587s. I think there's some correlation between the prevalence of these islands and the step number, which is why the Hypatia example took a larger number of steps to converge. However this shouldn't really affect the time per step, which is 0.07 at Flatiron vs 0.31 on Hypatia, but probably explained by the difference in the number of cores. Would be interested to see what you get out of NERSC @andreufont if you get things working!

Chris-Pedersen commented 2 years ago

Ok when running on the cores 40 nodes on Hypatia, we have 4100 steps in 951s, with a step time of 0.23, faster than cores24 but still far behind the Flatiron nodes.

andreufont commented 2 years ago

The performance at NERSC so far seems pretty poor, although I might not be running it properly. Just now I tried the small_test and it has only done 4000 steps in 22 minutes, or 1320s... This is on Cori, that in principle has 32 cores.

@Chris-Pedersen - do you have a script to make this pretty plot?

Chris-Pedersen commented 2 years ago

This performance looks similar to Hypatia, I guess the mystery is why flatiron is so fast..

Sure there are a load of methods of the sampler object to do this - the syntax is:

from lace.sampler import emcee_sampler
emcee_sampler.compare_corners([1],["Chain label"],plot_params=["Delta2_star","n_star"],subfolder="central_sim")

where the first argument is a list of chain numbers, the second list is a string of labels for the plot, plot_params are the params you want to plot (using code syntax, and will plot all free parameters if left as None), and subfolder is the subdirectory of the chains you want to access.