search

GRTLCollaboration / GRChombo

An AMR based open-source code for numerical relativity simulations.
BSD 3-Clause "New" or "Revised" License
82 stars 53 forks source link

Initialize MPI with thread support #237

Closed mirenradia closed 1 year ago

mirenradia commented 1 year ago

Although this is more correct (in terms of conforming to the MPI spec), I have not investigated the performance impact of this change. It is possible that launching with a higher level of MPI thread support may have a negative performance impact (though I believe this may only be significant for MPI_THREAD_MULTIPLE).

This fixes #199.

KAClough commented 1 year ago

It compiles, and when running on cosma7 with intel compilers, 18 nodes, 4 OMP threads per process, I find that it gives comparable speeds to a previous run (I actually find it gives slightly faster performance by 1 or 2 M/hr). However, sadly it does not fix the bug that the job does not exit, even though the values became Nan.

Edit: My setup was: number_procs = 126 threads = 4 mpi provided thread support = serialized simd width (doubles) = 8

I am using mpirun, the template jobscript for cosma7. I only get the nan output in the pout files, and not the error or output files that slurm produces.

mirenradia commented 1 year ago

@KAClough, even though they didn't solve your problem, if you're happy with these changes, would you be able to approve them so they can be merged?

KAClough commented 1 year ago

Sure, did you want to change the thread support to multiple or leave it as serial?

mirenradia commented 1 year ago

Sure, did you want to change the thread support to multiple or leave it as serial?

No, let's just leave it as MPI_THREAD_SERIALIZED.