Pool object

[sourestdeeds]

Im not too sure what the correct way to use multiple cpu's for sampling is. Currently my setup is:

nthreads=int(os.cpu_count())
with mp.Pool(processes=nthreads-1) as exec:
    sampler = NestedSampler(lnlike, prior, n_dims, bound=bound,  sample=sample, nlive=nlive, walks=100, slices=10, 
                           pool=exec, queue_size=nthreads)
    sampler.run_nested(maxiter=maxiter, maxcall=maxcall, dlogz=dlogz)
    results = sampler.results

  File "/Users/sourestdeeds/miniforge3/lib/python3.8/site-packages/dynesty-1.0.1-py3.8.egg/dynesty/dynesty.py", line 499, in NestedSampler
    live_v = np.array(list(M(ptform,
  File "/Users/sourestdeeds/miniforge3/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/sourestdeeds/miniforge3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/Users/sourestdeeds/miniforge3/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/Users/sourestdeeds/miniforge3/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/Users/sourestdeeds/miniforge3/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Retriever._run_dynesty.<locals>.prior_transform'

Do I need to use functools to pass this through properly?

[joshspeagle]

For pool objects to work properly within dynesty, you need to have the ability to pass through multiple functions and arguments due to how things are wrapped. I think functools should do the trick, but I'm not positive.

[sourestdeeds]

Thanks Josh! I may just use a different approach and parallel another way.

[shoepfl]

Hey, I am a littel confused. I tried to use multiprocessing to parallelize dynasty as there are also some examples in the issues here. However, I also gett a pickle error:

"AttributeError: Can't pickle local object 'DynestySampler.initialize..loglikelihood'"

I already implemented it with the manager of mulitprocessing (https://superfastpython.com/multiprocessing-pool-share-with-workers/) but still get the error.

Is multiprocessing supposed to work with dynasty? If not what are good arguments for pool? OpenMPI is not working for me as I am using it for Systems Biology together with amici which conflicts in many ways with OpenMPI

Thanks for any help in advance

[segasai]

Please open a new issue, and show the actual code. See the guidelines of how to report issues https://github.com/joshspeagle/dynesty/blob/master/.github/ISSUE_TEMPLATE/bug_report.md