Closed jwallwork23 closed 3 months ago
Small comment here with my experience, maybe irrelevant. I would adapt in serial by doing something like this in the adaptor function:
if COMM_WORLD.rank == 0:
arguments = f'--input temp_chk.h5 --output {output_fpath} --iteration {iteration}'.split()
subprocess.run(["mpirun", "-n", "1", "python", "adapt_script.py"] + arguments)
COMM_WORLD.barrier()
So I would have a separate script for adaptation that I would run in serial. If I did not do this, and I adapted under the if COMM_WORLD.rank == 0
statement (and not in a separate script that I would call using subprocess
), it would be extremely slow.
You recommended that I check the flame graph, but I gave up on using parallel so I never did it - but I can if we encounter this again / you think it's useful.
Thanks very much @ddundo, this is really useful info!
After some discussion, we are currently thinking this should actually happen on the Animate side. See https://github.com/pyroteus/animate/issues/71.
ParMmg doesn't support parallel adaptation in 2D. However, solving 2Ds PDEs in parallel is a standard activity of Firedrake. We should use Python
subprocess
to enable serial adaptation during a parallel run.