segasai
commented
1 year ago Hi,
I'll start with this example that shows that the uniform sampler does utilise all the cores if you have a heavy to compute function:
import numpy as np
import pytest
import dynesty
import multiprocessing as mp
import dynesty.pool as dypool
nlive = 1000
printing = True
ndim = 10
gau_s = 0.01
def loglike_gau(x):
for i in range(1000000):
1 + 1
return (-0.5 * np.log(2 * np.pi) * ndim - np.log(gau_s) * ndim -
0.5 * np.sum((x - 0.5)**2) / gau_s**2)
def prior_transform_gau(x):
return x
LOGZ_TRUTH_GAU = 0
LOGZ_TRUTH_EGG = 235.856
def test_pool_samplers():
# this is to test how the samplers are dealing with queue_size>1
rstate = np.random.default_rng(2)
with mp.Pool(10) as pool:
sampler = dynesty.NestedSampler(loglike_gau,
prior_transform_gau,
ndim,
nlive=nlive,
sample='unif',
pool=pool,
queue_size=10,
rstate=rstate,
first_update={
'min_eff': 90,
'min_ncall': 10
})
sampler.run_nested(print_progress=printing)
assert (abs(LOGZ_TRUTH_GAU - sampler.results['logz'][-1]) <
5. * sampler.results['logzerr'][-1])
if __name__ == '__main__':
test_pool_samplers()Regarding what's happening in your case. First there is not enough info at the moment to diagnose (one needs a reproduceable example). But the key difference of uniform sampling is that the proposal of points within ellipsoids is done in a single thread. And only later the likelihoods are being evaluated in parallel.
A possibility for what you're seeing is that the bounding ellipsoid you get for your problem is extreme (i.e. very elongated, much longer than the length of the cube), in that case most of the proposed points within the ellipsoid will be outside the cube and the code will try again etc, etc and that will all be done in a single thread, before the likelihood function is evaluated.
Another possibility is that your function is just too heavy to pickle and that dominates the overheads.
You can test the first hypothesis by lowering the threshold for the warning here https://github.com/joshspeagle/dynesty/blob/2d49cf9104e417a44ada7f2f1d013cbfee0acf01/py/dynesty/nestedsamplers.py#L705
The second hypothesis you can test by using dynesty.pool.Pool which eliminates the pickling overhead.
finally, if you cannot share a reproduceable example, you should at least share the exact way you call dynesty (with all the information, such as nlive, ndim etc. etc ) and ideally all the output from the problematic unif run.
MikhailBeznogov
Hello,
I have noticed a strange behavior of
dynestywhen parallelized (either with multiprocessing pool or MPI).Before the
first_updatecriterion is met, the sampler is alwaysunifand it is indeed parallelized. However, after thefirst_updatethe situation changes. If the requested sampler isunif, then, after thefirst_update, it is no longer working in parallel. To be more specific, multiprocessing pool is running, but only one process at a time is actually fully loaded, the other processes in the pool are idle. The number of likelihood evaluations per second also correspond to single-threaded computation.If the requested sampler is other than
unif(I have explicitly triedrwalkandrslice), then, after thefirst_update, the sampling is performed in parallel (and, judging by the CPU load, even more efficiently than before thefirst_update).Here are some examples. The first screenshot shows CPU load with the
unifsampler before and after thefirst_update(left black vertical line marks the start of the computation, the right one marks thefirst_update).The second screenshot shows the same for the
rslicesampler. Black vertical line marks thefirst_update. The difference can be clearly seen.Since the likelihood in my case takes only 1-2 ms to evaluate, I have also tried the same with huge "over-subscription" by setting
queue_sizeto much more than the actual number of threads (1600 vs 32). The results for theunifandrslicesamplers are demonstrated in the next two screenshots.As one can see, for the
unifsampler the results are qualitatively the same, although the performance has increased roughly 3 times (from ~10it/s just before thefirst_updateto ~30it/s). The same goes forrslicesampler, which become even more parallel-efficient.I have also checked the behavior of the
unifsampler when parallelized via MPI (without "over-subscription"). The CPU load in that case is always 100%, but the drop in the performance after thefirst_updateclearly indicates that the the situation is the same as with multiprocessing pool. Here is a part of the output "around" thefirst_update:The loss of performance after the
first_updateis clearly seen (both in number of iteration per second and in number of likelihood evaluations per second).Consequently, using the
unifsampler is very slow (unlessboundis set tononeand thefirst_updateis not executed, but this is extremely inefficient in combination with theunifsampler).P.S. All tests were done with the latest version, 2.1.0;
first_update={'min_eff':0.50}in all tests,slices=100for therslicesampler,bootstrap = 50. The tests above were performed withbound = 'multi', but the situation with other types of bounds is similar.