Closed tjgalvin closed 11 months ago
Looking a little closer at the beamcon_3D code I can see the parallelism approach is a little different to what I was expecting given what I knew of the beamcon_2D case.
As discussed offline, it looks like it might be a barrier that is holding things up. Working on a patch...
Also for completeness I started this project before seeing the light of Dask. I started with schwimmbad
, which couldn't hold up to the complexity of the 3D case. So, I ended up using bare mpi4py for that.
I have noticed that in beamcon_2D there is the capability to establish a pool object that is a little more versatile, allowing the user to establish MPI or multiprocessing parallelism.
I have noticed some weird interplay in
srun
and beamcon_3D, specifically around exceptions raised when beam log files do not exist. In the ASKAP pipeline beamcon_3D workers enter a zombie state and are only killed when the slurm job's walltime is reached.Is there a reason why mpi is the only parallelism mode made available in beamcon_3D?