search

JohannesBuchner / UltraNest

Fit and compare complex models reliably and rapidly. Advanced nested sampling.
https://johannesbuchner.github.io/UltraNest/
Other
143 stars 30 forks source link

subfolder with mpi4py #27

Closed stefanmarinus closed 3 years ago

stefanmarinus commented 3 years ago

Is it possible to have n independent subfolder outputs of the nested sampler when running with n cores in parallel? For example in the gauss.py code given in the documentation (https://johannesbuchner.github.io/UltraNest/performance.html#using-multiple-cores) the execution is mpiexec -np 4 python3 gauss.py. However, this leads to only one output folder even if in the ReactiveNestedSampler we have resume='subfolder'. What I would like to have is in this case 4 output folders of each running CPU, i.e. run1 for the results of CPU1 run2 for the results of CPU2,...

JohannesBuchner commented 3 years ago

What are you trying to achieve? Can you describe your underlying problem in more detail?

JohannesBuchner commented 3 years ago

If you want to analyse independent data sets, I usually pass the data file name as an argument to the script (receive in python with sys.argv[1]), and write to a logdir named after the data file (logdir=sys.argv[1] + '_out_fitmodelA').

Running parallel fits is then easy with ls data*|xargs --max-args=1 --max-procs=4 python3 myscript.py. It also makes it possible to write make scripts to resume computations (automatic variables / stems and parallelisation with -j4 are powerful). doit is a python alternative. Alternatively, you can also create new processes from within python, without using MPI.

resume='subfolder' creates a new folder for each run started. Runs share CPU power to distribute the load of finding a new independent point, but the run is coordinated by the processor with MPI rank 0. In other words, the run is not independent in the various CPUs.