bd-j / prospector

Python code for Stellar Population Inference from Spectra and SEDs
http://prospect.readthedocs.io
MIT License
159 stars 75 forks source link

Progress not displayed when using dynesty with MPI parallelization #353

Open tjtakys opened 1 week ago

tjtakys commented 1 week ago

Hi, I am using dynesty with MPI for parallel computation in my project, and I’ve encountered an issue with progress display. When running the code on a single core, the progress is displayed correctly as the computation proceeds. However, when running the same code using MPI for parallel processing, the progress is not shown during the calculation. Instead, the progress is only displayed either when the computation finishes or when it is interrupted midway. How can I display progress in real-time? Thanks in advance!

Environment: • dynesty version: 2.1.4 • mpi4py (Open MPI) version: 5.0.5 • Python version: 3.12.5 • OS: RockyLinux

Here is the relevant part of my code: (almost copy & paste from demo_mpi_params.py; the same issue occurs when running this demo file)

run_params["dynesty"] = True
run_params["emcee"] = False
run_params["optimize"] = False
run_params["nested_weight_kwargs"] = {'pfrac': 1.0}
run_params["nlive_init"] = 400
run_params["nlive_batch"] = 200
run_params["nested_dlogz_init"] = 0.1
run_params["nested_posterior_thresh"] = 0.1
run_params["nested_maxcall"] = int(1e7)
run_params["print_progress"] = True

mpi4py.rc.threads = False
mpi4py.rc.recv_mprobe = False

comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()

withmpi = size > 1

if withmpi:
        with MPIPool() as pool:
            if not pool.is_master():
                pool.wait()
                sys.exit(0)
            nprocs = pool.size
            print("Start fitting with MPI")
            output = fit_model(dict_obs, model, sps, pool=pool, queue_size=nprocs, lnprobfn=lnprobfn_fixed, **run_params)
    else:
        output = fit_model(dict_obs, model, sps, lnprobfn=lnprobfn_fixed, **run_params)

mpirun -np 8 python prospector_fit.py

bd-j commented 3 days ago

Hi,

I'm not sure why the output is different; my understanding is that different MPI implementations can have different ways of interacting with stdout or log files, especially with a progress bar. At the moment we have no plans to provide detailed support for MPI, but I note that the sampler backends are used in a much simpler manner in the upcoming v2.0, which should hopefully standardize some of these issues.