liruipeng
commented
5 years ago Dear
hypredevelopers!The
deal.IIcommunity is currently making an effort to implement parallel hp-adaptive finite element methods. We were preparing a particular benchmark to verify our parallel implementation, which is based on one of thedeal.IItutorial step step-27 in which sequential hp-adaptive FEM has been introduced.However, a very certain corner configuration of our scenario yields to some errors triggered by
hypre, or more specifically, byBoomerAMG. We use a backend toPETScas our parallel linear algebra library, which providesBoomerAMGas our algebraic multigrid preconditioner.We could only reproduce this issue with exactly 20
MPIprocesses assigned to a total of 49152 cells with 198144 degrees of freedom. The error occurs in the very first cycle of consecutive refinements, in which all cells were assigned a Lagrangian finite element with quadratic base functions.Using a different preconditioner or none at all does not trigger the error. Neither does a different amount of
MPIprocesses, nor a different number of global mesh refinements (which yields this particular amount of cells). However, I haven't tried different finite elements yet. With parallel debugging, I encountered the following issues:Five processes reported the following (with
petsc-3.9.1):Four processes reported the following (with
petsc-3.9.1):Additionally, when using versions of
PETScprior to3.8.4, error messages likely to the following appear.bash-4.2$ mpirun -np 20 step-27new Running with PETSc on 20 MPI rank(s)... Cycle 0: Number of active cells : 49152 Number of degrees of freedom: 198144 Number of constraints : 3272 Out of memory trying to allocate 1451474080 bytes Out of memory trying to allocate 1451474080 bytes Out of memory trying to allocate -1938871136 bytes Out of memory trying to allocate -1938871136 bytes Out of memory trying to allocate 1073649184 bytes Out of memory trying to allocate 1073649184 bytes Out of memory trying to allocate 595867104 bytes Out of memory trying to allocate 595867104 bytes -------------------------------------------------------------------------- mpirun noticed that process rank 1 with PID 36483 on node down exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------We have been discussing this topic in dealii/dealii#8565. We tried to locate the error using several
git bisectsessions, and the problem seems to have been introduced in one of the following commits f4e70c8...a08bfa5. The test case completes with f4e70c8 and fails with a08bfa5 for the first time.
Hi, @marcfehling,
Trying to understand something here: f4e70c8 was newer than a08bfa5. (01/2016 v.s. 12/2015). Does the newer one f4e70c8 work while a08bfa5 not?
We were not able to narrow the selection of commits down any further, since we were not able to build
hypresuccessfully for any of them. I know that, since this is a very special corner case, it will most probably not be easy to find the source of the error, but maybe you have a very first idea on where this error may come from regarding thegdboutput above. I hope that we could make a joint effort to tackle this problem together.Please let me know if you need further information on our scenario, or if you would like to work with the aforementioned
deal.IItestcase and need assistance with it. You'll find the sourcecode on this branch. Pardon me if it you find it untidy - it is in some kind of work-in-progress state. If desired, I will try to boil it down to a minimal working example.
marcfehling
bangerth
Dear
hypredevelopers!The
deal.IIcommunity is currently making an effort to implement parallel hp-adaptive finite element methods. We were preparing a particular benchmark to verify our parallel implementation, which is based on one of thedeal.IItutorial step step-27 in which sequential hp-adaptive FEM has been introduced.However, a very certain corner configuration of our scenario yields to some errors triggered by
hypre, or more specifically, byBoomerAMG. We use a backend toPETScas our parallel linear algebra library, which providesBoomerAMGas our algebraic multigrid preconditioner.We could only reproduce this issue with exactly 20
MPIprocesses assigned to a total of 49152 cells with 198144 degrees of freedom. The error occurs in the very first cycle of consecutive refinements, in which all cells were assigned a Lagrangian finite element with quadratic base functions.Using a different preconditioner or none at all does not trigger the error. Neither does a different amount of
MPIprocesses, nor a different number of global mesh refinements (which yields this particular amount of cells). However, I haven't tried different finite elements yet. With parallel debugging, I encountered the following issues:Five processes reported the following (with
petsc-3.9.1):Four processes reported the following (with
petsc-3.9.1):Additionally, when using versions of
PETScprior to3.8.4, error messages likely to the following appear.We have been discussing this topic in https://github.com/dealii/dealii/issues/8565. We tried to locate the error using several
git bisectsessions, and the problem seems to have been introduced in one of the following commits https://github.com/hypre-space/hypre/compare/f4e70c8...a08bfa5. The test case completes with https://github.com/hypre-space/hypre/commit/f4e70c8 and fails with https://github.com/hypre-space/hypre/commit/a08bfa5 for the first time. We were not able to narrow the selection of commits down any further, since we were not able to buildhypresuccessfully for any of them.I know that, since this is a very special corner case, it will most probably not be easy to find the source of the error, but maybe you have a very first idea on where this error may come from regarding the
gdboutput above. I hope that we could make a joint effort to tackle this problem together.Please let me know if you need further information on our scenario, or if you would like to work with the aforementioned
deal.IItestcase and need assistance with it. You'll find the sourcecode on this branch. Pardon me if it you find it untidy - it is in some kind of work-in-progress state. If desired, I will try to boil it down to a minimal working example.