search

AlecThomson / RACS-tools

Useful scripts for RACS
BSD 3-Clause "New" or "Revised" License
12 stars 7 forks source link

multiprocessing pool for beamcon_3D #43

Closed tjgalvin closed 11 months ago

tjgalvin commented 1 year ago

I have noticed that in beamcon_2D there is the capability to establish a pool object that is a little more versatile, allowing the user to establish MPI or multiprocessing parallelism.

I have noticed some weird interplay in srun and beamcon_3D, specifically around exceptions raised when beam log files do not exist. In the ASKAP pipeline beamcon_3D workers enter a zombie state and are only killed when the slurm job's walltime is reached.

Is there a reason why mpi is the only parallelism mode made available in beamcon_3D?

tjgalvin commented 1 year ago

Looking a little closer at the beamcon_3D code I can see the parallelism approach is a little different to what I was expecting given what I knew of the beamcon_2D case.

AlecThomson commented 1 year ago

As discussed offline, it looks like it might be a barrier that is holding things up. Working on a patch...

AlecThomson commented 1 year ago

Also for completeness I started this project before seeing the light of Dask. I started with schwimmbad, which couldn't hold up to the complexity of the 3D case. So, I ended up using bare mpi4py for that.