Open fdkong opened 2 years ago
roystgnr
commented
2 years ago We're fairly confident we've hit bugs in MVAPICH before. Nothing that manifests like this exactly ... but I absolutely would love to know if the same code runs or fails if run on sawtooth with, say, OpenMPI.
fdkong
commented
2 years ago The test from has the problem when using mvapich2/2.3.3-gcc-9.2.0
Occasionally, the same test ran just fine with mvapich2/2.3.5-gcc-8.4.0.
We need to dig into MPI to figure out what was the root cause of that. And then we could improve parallel_vector_push/pull by using more-robust MPI APIs.
roystgnr
commented
2 years ago There is such a thing as less robust MPI APIs but they're not the problem here. Either we've got some kind of race condition or mvapich has got some kind of bug. From what @friedmud was telling us about leaks, I suspect it's the latter.
But at this point it seems clear we can't just hope to mostly be unscathed by it. Either we need to demonstrate that it's a real implementation problem and pressure them for a fix, or we need to switch to a different MPI implementation (this might not be a good option? IIRC there are some fabrics where MVAPICH is at least twice as fast as competitors) or we need to switch to APIs that aren't subject to leaks (this might be the worst option of all, on large runs? our old implementation here was asymptotically slower as the number of processors increases)
fdkong
commented
2 years ago we need to switch to a different MPI implementation (this might not be a good option? IIRC there are some fabrics where MVAPICH is at least twice as fast as competitors)
I would like that MOOSE works for all reasonable MPI implementations.
At this point, I still think the bug(s) can come from MOOSE, TMPI, or MPI. I do not want to say it is the fault of MVAPICH because a ton of other packages are working well on MVAPICH.
But anyway, we will figure that out once we dig into the bottom..
roystgnr
commented
2 years ago The most relevant parts from Slack discussion a few months ago:
Derek Gaston:
The "eager sends" ARE messed up in mvapich - and they refuse to fix it (I've been talking to them for a year about it)... it could easily run out of eager send memory. We may want to try disabling eager sends to see if that fixes it.
Roy Stogner:
That sounds almost too good to be true if it works ... but I am seeing a lot of MPIDI_CH3_PKT_EAGER_SEND in those error messages. And "Recv desc error" and "completion with error" sure sound like they're coming from a lower level than us.
Derek Gaston:
Yes - it could literally be the eager send pool running out of device memory.
(On the infiniband cards)
Basically: mvapich is trying to do too many eager sends (it shouldn't do any since we're doing asynchronous anyway!)
Derek Gaston:
This may take some tinkering. Looking at the messages that were failing to receive correctly - it looks like they may be just below the default eager cutoff (which is normally 12KB). What we could try to do is lower the eager limit using: export MV2_IBA_EAGER_THRESHOLD=2k export MV2_VBUF_TOTAL_SIZE=2k You will want those in your launch script
To give some idea of what's going on: mvapich attempts to send really small messages via a different communication algorithm called the "eager" algorithm. The way it works is that it copies your data into an internal buffer (on the infiniband card if it can) and then tries to push the data. As soon as the copy is done mvapich reports back to the program that the send is complete (even if it's not). This means that a program can stack up millions of tiny messages to send without knowing that they are just piling up and not getting sent. This whole idea is antiquated... and came about because it "sped up" old programs that use blocking MPI calls. By acting like the data is "sent" MPI can let the program continue to operate... somewhat overlapping communication and computation (and gaining efficiency). But - since we do everything asynchronously anyway... we just end up with a whole bunch of unnecessary copying and tiny messages filling up memory. So: if we lower the limit... then only really tiny messages would be sent this way. We could set the above to 0k to turn off eager sends all together. I have no idea what that would do to overall efficiency of our code (I'm sure there are still many places where we benefit from eager sends in libMesh/PETSc/Hypre)
Derek Gaston:
Ok - so this is about mvapich running out of vbuf memory because of the way they do eager sends and "leak" the buffer memory (It's not actually leaking... it's just never freed until the end of the simulation). We need to try OpenMPI (or some other MPI) to double check that it's fine. Like I say - I've been talking to the mvapich people about this for nearly a year. I even created a simple gist that shows the issue: https://gist.github.com/friedmud/9533d5997f06414c25f8c5c57a1eaf37
Derek Gaston:
They have a vbuf pool - that they continually "check out" memory from... but never return it to.
It is only returned during MPI_Finalize.
(So, it is not leaked... it is kept track of... it's just not returned to the pool and not reused)
Roy Stogner:
Ha!
"Not returned and not reused" is the definition of "leaked", to me. Doing it on purpose is an aggravating factor, not a mitigating one.
Derek Gaston:
The reason that no one else sees this is because they are not doing completely asynchronous algorithms like we are. I guess I should clarify that this only happens with isends. With regular sends - the eager memory is "freed"
Roy Stogner:
You can demonstrate this in about a hundred lines of code and they've still dragged their feet on fixing it for a year?
That's awful.
Derek Gaston:
They continue to deny there is a problem
(and drag their feet - sometimes months between communications - and they took the conversation off of their regular user list because they said it was generating too much noise)
So: let's try using a different MPI implementation and see if that clears it up.
Roy Stogner:
There's got to be a better way to frame this for the mvapich people... Instead of a single batch of sends, would it be possible for your test code to send batch after batch in such a way that other MPI implementations can run indefinitely but theirs will eventually run out of pool memory and die?
Derek Gaston:
I mean - just put a loop around my code
Roy Stogner:
And that'd be all it would take to do it? They're just never reusing that memory? I can come up with no words that would be both politic and accurate here.
Larry Aagesen:
So I found that with more memory per core, both the openmpi run and a run with the buffer setting size set to 128k ran all the way to completion
roystgnr
commented
2 years ago So based on that,
I would like that MOOSE works for all reasonable MPI implementations.
I would too. But I don't think "if you keep using asynchronous I/O indefinitely you will eventually die" qualifies as reasonable.
At this point, I still think the bug(s) can come from MOOSE, TMPI, or MPI
I certainly wouldn't be surprised if we had bugs in addition. But this certainly seems to be a bug, not in MPI, but in that one particular MPI implementation.
I do not want to say it is the fault of MVAPICH because a ton of other packages are working well on MVAPICH.
That's the right prior to have, isn't it? But take a look at Derek's gist. Just a couple hundred lines, no obvious bugs.
roystgnr
commented
2 years ago Your test case runs fine for me with both opt and dbg modes. I'm using the newest libMesh, though, and I'm trying it locally rather than on sawtooth.
maeneas
commented
2 years ago Trying on Sawtooth now...
maeneas
commented
2 years ago duplicated issue on Sawtooth
maeneas
commented
2 years ago export MV2_IBA_EAGER_THRESHOLD=2k export MV2_VBUF_TOTAL_SIZE=2k has no impact on issue on Sawtooth.
maeneas
commented
2 years ago mvapich team is looking into this now
friedmud
commented
2 years ago Relatedly... see if you can get them to fix the memory "leak" they have with isend. See my gist here: https://gist.github.com/friedmud/9533d5997f06414c25f8c5c57a1eaf37
It's not a true leak because the memory is not "lost" - it's just that a new temporary buffer gets created for each isend and doesn't get released until MPI_finalize.
I tried to talk to them about it but couldn't get them to do anything about it.
friedmud
commented
2 years ago I believe that the above buffer issue is the root of the problem we're seeing here...
permcody
commented
2 years ago @maeneas - Do we have TotalView available these days for MPI debugging? I'm not sure if that would help here or not, but we could support purchasing licenses if it makes sense.
maeneas
commented
2 years ago export MV2_USE_SHARED_MEM=0 solves the hang problem. Still investigating further.
maeneas
commented
2 years ago We don't have any totalview licenses available anymore. I just debug with gdb anyways.
maeneas
commented
2 years ago Adjusting MV2_SMP_SEND_BUF_SIZE has some impact on whether this bug shows up. Setting MV2_USE_SHARED_MEM=0 solves the problem uniformly in all tests but probably negatively impacts performance.
permcody
commented
1 year ago @roystgnr, @friedmud - This is the same issue that Patrick is hitting last week. We could try these workarounds again. Also @loganharbour
Bug Description
There is a bug that possibly is in MVAPICH, TMPI, or MOOSE. For some applications during the mesh partitioning, all ranks are waiting for messages, and computations are hanging.
Steps to Reproduce
This can be reproduced on Sawtooth.
1)
module load use.moose PETSc2)
git clone moose3) build libmesh:
./script/update_and_rebuild_libmesh.sh4) complie moose:
make -j244) run the following using an interactive node:
mpirun -n 44 ../moose_test-opt -i master.iInput-files are too large to load them here. Please contact the ticket creator to get the input files that will reproduce the issue.
Impact
Users can not run large scale simulations INL HPC.