I was able to get the test_mpi.py example to work, but I have not been able to get the chain manager to work when trying to use it inside a function or importing it. This functionality is desirable so that people can use custom run files or make other packages that use zeus as a dependency (which is what I am doing).
I am providing example files and how to reproduce the behaviour.
In the attached files 4.0MPITest.zip
This works:
mpiexec -n 5 python test_mpi.py
These below 3 do not work. They run, but they do not do have the correct behavior, they do not actually seem to initiate walkers and they do not export the expected ".npy" files.
The below 4th case runs for a bit , then 'stalls' or crashes with message that it can't pickle the object. This is when using zeus 2.0 (I don't think the below 4th case crashed before zeus 2.0, but I am not sure anymore).
mpiexec -n 5 python test_mpi_by_function4.py
Proposed solution: Unknown. But it would be nice if a clever way can be found to allow chain manager to work inside functions and when imported elsewhere. At present, it is rather puzzling to me that it doesn't work.
At the moment I am just going to use MPI externally. For example, if I have 8 processors, I will do 8 entirely separate EnsembleSampling and then combine the results. This is not ideal for a variety of reasons. I thought about generating a custom python file each time and running mpi from the command line, but that is not a suitable solution for me.
I was able to get the test_mpi.py example to work, but I have not been able to get the chain manager to work when trying to use it inside a function or importing it. This functionality is desirable so that people can use custom run files or make other packages that use zeus as a dependency (which is what I am doing).
I am providing example files and how to reproduce the behaviour.
In the attached files 4.0MPITest.zip This works:
mpiexec -n 5 python test_mpi.pyThese below 3 do not work. They run, but they do not do have the correct behavior, they do not actually seem to initiate walkers and they do not export the expected ".npy" files.
The below 4th case runs for a bit , then 'stalls' or crashes with message that it can't pickle the object. This is when using zeus 2.0 (I don't think the below 4th case crashed before zeus 2.0, but I am not sure anymore).
mpiexec -n 5 python test_mpi_by_function4.pyProposed solution: Unknown. But it would be nice if a clever way can be found to allow chain manager to work inside functions and when imported elsewhere. At present, it is rather puzzling to me that it doesn't work.
At the moment I am just going to use MPI externally. For example, if I have 8 processors, I will do 8 entirely separate EnsembleSampling and then combine the results. This is not ideal for a variety of reasons. I thought about generating a custom python file each time and running mpi from the command line, but that is not a suitable solution for me.