Problem getting multi-threading to work in dynesty

[shantanu9847]

Dear all, I seem to have a similar problem as discussed in https://github.com/joshspeagle/dynesty/issues/100 where one core gets 100% CPU and the remaining cores show only a small fraction. So as suggested in the above github link I used bootstrap=0 , but I am still having the same problem. I have attached the script. The top output shows

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3766 iith 20 0 1145648 171620 42988 S 98.0 0.3 15:07.64 python3
2164 iith 20 0 6205688 309168 122084 S 12.9 0.5 1:23.15 gnome-shell 4056 iith 20 0 1186728 179236 11756 S 3.6 0.3 0:06.00 python3
4063 iith 20 0 1198880 190608 11104 S 3.6 0.3 0:06.10 python3
4066 iith 20 0 1198536 191112 11944 S 3.6 0.3 0:05.98 python3
1939 iith 20 0 2272044 154884 83124 R 3.3 0.2 0:14.80 Xorg
4061 iith 20 0 1198632 190956 11684 S 3.3 0.3 0:05.93 python3
4062 iith 20 0 1199024 189076 9424 S 3.3 0.3 0:06.04 python3
4064 iith 20 0 1198916 189568 10024 S 3.3 0.3 0:06.05 python3
4065 iith 20 0 1198776 190952 11548 S 3.3 0.3 0:05.93 python3
4057 iith 20 0 1197856 187912 9424 S 3.0 0.3 0:05.98 python3
4060 iith 20 0 1198688 190740 11420 S 3.0 0.3 0:06.09 python3
2508 iith 20 0 793844 37468 27424 R 2.6 0.1 0:03.62 gnome-term+ 4058 iith 20 0 1198700 188752 9424 S 2.6 0.3 0:05.98 python3
4059 iith 20 0 1199240 191816 11944 S 2.3 0.3 0:05.89 python3

As you can see %CPU is very small among the other cores. The only difference between the script on the github page and the one attached is that I used multiprocess module instead of multiprocessing since I got the same error as on this page below with multiprocessing https://stackoverflow.com/questions/72766345/attributeerror-cant-pickle-local-object-in-multiprocessing

Anyhow any hints to enable #the script to use 100% cpu on each of the other cores would be very much appreciated. Thanks shantanu

The relevant parts of the code are

from dynesty import plotting as dyplot
from dynesty import DynamicNestedSampler
from dynesty.utils import resample_equal
from multiprocess import Pool,cpu_count

def dynesty_sample(pta):
    prior_transform = prior_transform_fn(pta)
    with Pool(cpu_count()-1) as executor:
        sampler = dynesty.NestedSampler(pta.get_lnlikelihood, prior_transform, len(pta.params),bound='multi',nl
ive=1024,pool=executor,queue_size=cpu_count(),bootstrap=0)
        sampler.run_nested(dlogz = 0.1,print_progress=False)
    res = sampler.results
    return res

[segasai]

Hi,

You need to provide more information, like what dynesty version you're using, what's the dimensionality of the problem, how long is to evaluate your likelihood function, ideally show output from dynesty run, etc.

Also, if I had to guess your likelihood function together with data attached to it is so heavy/takes so long to pickle/dill that it is the limiting factor here.

[shantanu9847]

I am using v1.2.3 version of dynesty and python3.8. The model has two free parameters

[segasai]

I am not sure we can help if you don't provide enough information (like I suggested in the previous post)

As I said, I suspect that your function takes very short amount of time to evaluate and/or very takes very long time to get pickled therefore the parallelisation will not help you much there.

Also in low dimensions when unif sampler is being used ( and the bounding ellipsoid is very large), the main limitation can be occasionally generating those samples within the ellipsoid bound, and that part is not parallelized.

[segasai]

I'll close this. Feel free to reopen, if there is still an issue and you want to provide more info.