search

JohannesBuchner / BXA

Bayesian X-ray analysis (nested sampling for Xspec and Sherpa)
https://johannesbuchner.github.io/BXA/
GNU General Public License v3.0
57 stars 19 forks source link

How to use more than one core for computation? #8

Closed alexkolo closed 5 years ago

alexkolo commented 5 years ago

At the moment BXA (actually pymultinest) only uses one core. How can I make it use more than one?

JohannesBuchner commented 5 years ago

Install MPI, recompile Multinest to make sure you have libmultinest_mpi.so. Install the python package mpi4py. Then run with "mpiexec -np 4 python3 myprogram.py".

alexkolo commented 5 years ago

I was afraid you will say that. The problem is that "myprogram.py" was not designed to be running in parallel and adjusting it is quite the effort. I hoped that one could just use Pool from multiprocessing for just executing a BXA function but I guess that's not an option :/ PS: Thx for the super quick feedback!

alexkolo commented 5 years ago

I managed to rewrite my code in order to make use of the multiprocessing function of MulitNest with the help of mpiexec. One important step was that I needed to (re)build MulitNest with the exactly same MPI package, which is used by the python packaged mpi4py. (in my case that would be cmake -DCMAKE_PREFIX_PATH=~/Software/Science/anaconda3/envs/science .. && make ) I also optimized the BXA code (editing standard_analysis and nested_run functions), since only when executing pymultinest.run I need more than one process. I placed a MPI.COMM_WORLD.Barrier() just before executing pymultinest.run to make sure that all processes are at the same step but I'm not 100% sure if that's really necessary.

I'm still rather new to python, made my transition from IDL less than a year ago. Hence, I'm yet not familiar with forking and updating code on GitHUB. Otherwise, I would have uploaded my version of BXA here. In any case, I would like to use the opportunity here to thank you for building such an amazing tool. It's super convenient and handy.

However, I notice that the computation time doesn't reduce linear with the number of process used. For instance, when using 7 cores/processes the time is only reduced by a factor of 3 in comparison to using one core/process. Is this to be expected?

JohannesBuchner commented 5 years ago

Glad you got it working.

The steps to contribute changes are:

If this is too complicated, you can also email me (or attach here) the output of git diff.

JohannesBuchner commented 5 years ago

And thank you for the compliments. I still feel BXA should be 1) easier to install, 2) have more functionality (for visualization, model checking) and 3) be easier to understand for new-comers. If you have any suggestions, contributions and changes are very welcome.