What is the correct way of running dynesty on many CPUs using the "pool" argument?

[3fon3fonov]

Generally, I am able to run "dynesty" with multiprocessing, but sometimes I am reaching a bottleneck when all CPU pools must wait for the process to finish. There must be a way to do it with a "while" loop or something??? Without going to great details I usually do something like:

from pathos.pools import ProcessPool as Pool

dynesty_samp = "rwalk"
print_progress = True
N_threads = 40
stop_crit = 0.01
Dynamic_nest = True
ns_bound = "multi"
ns_pfrac = 1.0
ns_use_stop = True
ndim = 30 # for example
nlive=1000

thread = Pool(ncpus=N_threads)

sampler = dynesty.DynamicNestedSampler(partial_func, prior_transform, ndim, pool = thread,
                                       queue_size=threads, sample = dynesty_samp, bound = ns_bound) # nlive=nwalkers,

sampler.run_nested(print_progress=print_progress,dlogz_init=stop_crit,nlive_init=nwalkers, 
maxiter = ns_maxiter, maxcall = ns_maxcall,use_stop = ns_use_stop, wt_kwargs={'pfrac': ns_pfrac})
thread.close()
thread.join()
thread.clear()

but on heavy modeling schemes, I often get only one thread/cpu working, whereas the rest are idling. Perhaps the runs are as independent as I would like to?

Can you help with some wisdom?

p.s. pathos is a wrapper around multiprocess but is far better because it allows you to pickle almost everything.

Thanks! Trifon

[joshspeagle]

Internally, dynesty essentially just calls pool.map, which does the really dumb thing and has the drawback which is exactly what you're describing. There 100% are better ways to do this so that points can get evaluated on the fly while the queue is always filled, but unfortunately dynesty does not support those options.

[3fon3fonov]

Thanks! Let's keep this open a while if someone wants to share experience.

[joshspeagle]

Just closing this since there hasn't been any updates over the past few months. Happy to reopen if there are still issues to be worked out.