Closed JanKleine closed 5 years ago
Hi, what MPI version and fabric are you using? We have seen some implementations not behaving correctly.
I'm using openmpi 3.1.3 with InfiniBand. Thanks for the quick response.
Ok. So that seems fine. And what version of the OFED driver are you using? The regular Linux-kernel one or the Mellanox-optimised?
Also, what transport library are you using in OpenMPI?
We recommend psm
and not psm2
. That is running with --mca btl vader,self --mca mtl psm
.
Ok. So that seems fine. And what version of the OFED driver are you using? The regular Linux-kernel one or the Mellanox-optimised?
I think the Mellanox-optimised version
We recommend
psm
and notpsm2
. That is running with--mca btl vader,self --mca mtl psm
.
I will try that
Ok. So that seems fine. And what version of the OFED driver are you using? The regular Linux-kernel one or the Mellanox-optimised?
I think the Mellanox-optimised version
Right, then that is likely the issue. Their curent driver hangs if too many asynchronous communications are in-flight at a given point in time. SWIFT makes extensive use of this mechanism so you may be facing this issue here.
Would removing the Mellanox driver fix the issue?
I can only speculate as I have never seen this issue on machines where we have control over things but it may help.
Otherwise, trying a different mtl
in OpenMPI might help (instead of changing driver).
Right, then that is likely the issue. Their curent driver hangs if too many asynchronous communications are in-flight at a given point in time. SWIFT makes extensive use of this mechanism so you may be facing this issue here.
I seem to get the same problem on a system with the regular driver.
Did you try changing the mtl
to psm
?
Hi Jan,
Have you had some luck with the code?
I'm having some trouble as the OpenMPI version I'm using apparently doesn't support psm and installing it with psm is a little problematic at the moment, but I'm still on it.
I'm also using Slurm as job scheduler, I forgot to mention that earlier. I hope that that is not interfering with anything.
Did you try changing the
mtl
topsm
?
~using psm
did not seem to resolve the issue.~
Edit: I made another mistake while running. Using psm
does seem to resolve the problem
I was trying to run the Eagle 50 example (swiftsim/examples/EAGLE_low_z/EAGLE_50) as part of preparing for a competition. All tests pass, but the simulation seems to be stuck before even starting properly (cpu utilisation goes to 1-2 cores per rank, and nothing changes after running like this for 12h). I used
--cosmology --hydro --self-gravity --stars --threads=16 -n 64 --pin
full output
Sometimes it gets stuck earlier (at
main: from t=1.276e-02 until t=1.413e-02 with 4 ranks...
) but I've never gotten it further than what's shown above.What am I doing wrong?