I got error when passing in multiprocessing pool to `kliff.uq.MCMC`

[yonatank93]

When we instantiate kliff.uq.MCMC, it instantiates sampler from ptemcee (ptemcee.Sampler) or emcee (emcee.EnsembleSampler). In both emcee and ptemcee, we can pass it additional keyword arguments, such as the random state and multiprocessing pool, and I set it so that kliff.uq.MCMC can do the same. (Pretty much, I passed in the entire kwargs to either ptemcee.Sampler or emcee.EnsembleSampler).

However, we have a problem when we declare the multiprocessing pool before instantiating kliff.uq.MCMC. The error message I got was: AttributeError: Can't get attribute 'loglikelihood_fn' on <module 'kliff.uq.mcmc' from 'kliff/kliff/uq/mcmc.py'> (See this example to reproduce this issue)

What I understood from this discussion, this error is because the loglikelihood_fn in kliff.uq.MCMC, which is used in the parallel run, is defined after I declare the pool (see these lines).

As a work around, I was thinking about adding a property pool with its setter to set the multiprocessing pool for the sampler, and force users to set the pool after instantiating kliff.uq.MCMC. For example,

sampler = kliff.uq.MCMC(...)
sampler.pool = multiprocessing.Pool(nprocesses)

Do you have any suggestion on how to deal with this issue?

[mjwen]

I ran the example you've linked, but got different types of error:

  raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I am running on Mac, are you running on Linux?

[yonatank93]

I am running this on Linux. For your error, I found this article that might be related.

[yonatank93]

I have a working solution on the parallelization part, that also allows us to use MP for cost evaluation and MP or MPI for sampling simultaneously. I opened PR #66 for this.

[mjwen]

Resolved at #66.