TypeError: can't pickle SwigPyObject objects, when I call the dynesty in the pipeline

[yaojian95]

Hi, I recently wrap the dynesty functions, with Pool to get things parallelled, into a pipeline such as

Class Loglike(object):
    def __init__():
    ....
    def dynesty(self, ....):

        with Pool(cpu_count()-1) as executor:

            sampler = dynesty.NestedSampler(self.log_likelihood, self.prior, npara, nlive=400, pool=executor, queue_size=cpu_count(), bootstrap = 0)
            sampler.run_nested()
            results = sampler.results

        return results

but get the error.

TypeError: can't pickle SwigPyObject objects

I realized that the Pool from multiprocessing can only work for top-level functions and the error happens when I wrap the dynesty sampler into a class, could you give some suggestions on how to solve this?

[joshspeagle]

Dealing with what pools can and can't pass back is always a super pain. I notice that you are defining dynesty as a function call within a class while also importing it. Could this be causing some of the issues?

[yaojian95]

I tried to change the function name in the class and the error remains...

[joshspeagle]

I'm not an expert in dealing with pool pickling issues, unfortunately, so I don't have too much specific advice I can give here other than maybe trying to define your functions directly rather than as features of a class and/or just assigning the pool at a higher level. Sorry I can't provide more help!

[yaojian95]

Thanks for your reply anyway. I will try to do some research on this topic and let you know if I have some progress.