search

hypre-space / hypre

Parallel solvers for sparse linear systems featuring multigrid methods.
https://www.llnl.gov/casc/hypre/
Other
675 stars 187 forks source link

Error code 256 #344

Closed bensworth closed 3 years ago

bensworth commented 3 years ago

I have a script I'm running through MFEM that worked a few months ago and I just went to run it again and get the errors

Verification failed: (!err) is false:
 --> Error during solve! Error code: 256
 ... in function: virtual void mfem::HypreSolver::Mult(const mfem::HypreParVector&, mfem::HypreParVector&) const
 ... in file: /Users/southworth/Software/mfem/linalg/hypre.cpp:2690

Searching through hypre, it appears this means the coarse-grid solve did not converge. Did something change here? I am using AIR, classical AMG interpolation, and Falgout coarsening. I am not setting the coarse grid solver. Any help would be appreciated, not quite sure where to look here.

liruipeng commented 3 years ago

Hi @bensworth , Do you know what's the coarse-grid solver used? The default is Gauss elimination. I assume something else is used here.

bensworth commented 3 years ago

I'm not setting anything for the coarsest grid solver. To debug I've gotten to where I comment out all specific hypre set functions, and have tried w/ or w/o, e.g., HYPRE_BoomerAMGSetRelaxType(*amg, 3). Is there a way to specifically set or check the coarset grid solver?

ulrikeyang commented 3 years ago

You can set the coarse grid solve using HYPRE_BoomerAMGSetCycleRelaxType(amg, rlx_type, 3); Generally the default should be 9 for Gaussian elimination, but under certain circumstances (e.g. if the coarse grid is really large or it stops coarsening early) it could end up being Jacobi (potentially without a weight). But it still should only do 1 sweep of that or whatever you set. You can set HYPRE_BoomerAMGSetCycleNumSweeps(amg, num_seeps, 3), if you want more than 1. So, I don’t understand the statement that the coarse grid solve doesn’t converge. We generally don’t check for coarse grid convergence.

From: Ben Southworth @.> Sent: Thursday, April 22, 2021 7:17 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Assign @.> Subject: Re: [hypre-space/hypre] Error code 256 (#344)

I'm not setting anything for the coarsest grid solver. To debug I've gotten to where I comment out all specific hypre set functions, and have tried w/ or w/o, e.g., HYPRE_BoomerAMGSetRelaxType(*amg, 3). Is there a way to specifically set or check the coarset grid solver?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/344*issuecomment-824881824__;Iw!!G2kpM7uM-TzIFchu!jLLt-nPat92Aa6_AzgzRomgpLbb97K7KjMiTaQVZIKBtD9wOPPIAWmR6tHmLNdOA$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLPAX77M7AFO22IEMPTTKAVXRANCNFSM43LIP4KQ__;!!G2kpM7uM-TzIFchu!jLLt-nPat92Aa6_AzgzRomgpLbb97K7KjMiTaQVZIKBtD9wOPPIAWmR6tBBhl9UI$.

bensworth commented 3 years ago

If I set HYPRE_BoomerAMGSetCycleRelaxType(*AMG_solver, 9, 3); I still get Error code: 256 from the hypre solver. Will try to make a MWE.

ulrikeyang commented 3 years ago

Can you set it to 18 to see what happens? What is a MWE?

From: Ben Southworth @.> Sent: Thursday, April 22, 2021 7:28 AM To: hypre-space/hypre @.> Cc: Yang, Ulrike Meier @.>; Assign @.> Subject: Re: [hypre-space/hypre] Error code 256 (#344)

If I set HYPRE_BoomerAMGSetCycleRelaxType(*AMG_solver, 9, 3); I still get Error code: 256 from the hypre solver. Will try to make a MWE.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/hypre-space/hypre/issues/344*issuecomment-824890644__;Iw!!G2kpM7uM-TzIFchu!i8ktf_yHYlMzTFi22lODI1Gscdrd4RSoBTlVHPhkjSAoife0P-XPm3O302OOtr6C$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AD4NLLLQ66XZA65CVXIU5DDTKAXADANCNFSM43LIP4KQ__;!!G2kpM7uM-TzIFchu!i8ktf_yHYlMzTFi22lODI1Gscdrd4RSoBTlVHPhkjSAoife0P-XPm3O30zNZBYgD$.

bensworth commented 3 years ago

Same thing with 18. MWE = minimal working example. This is being used with some additional pieces of code, but might be easier to narrow down if I can get it self-contained in a script

oseikuffuor1 commented 3 years ago

@bensworth are you doing multiple solves with the AMG solver? If so, can you confirm that this error code happens at the very first solve? Also, if you are using AMG as a preconditioner, can you confirm that you have set the convergence tolerance to zero? Sometimes this error code is thrown if the tolerance is not reached. In that case, a subsequent call to the solver will fail with this error code. Setting the tolerance to zero tells AMG to use do the prescribed number to iterations without checking for convergence on the tolerance.

bensworth commented 3 years ago

Yep, I am using AMG as a preconditioner for time integration, error happens on the first step. I had set the tolerance to 1e-12, but set it to 0 and get the same error code.

oseikuffuor1 commented 3 years ago

Ok just to confirm that you set the AMG preconditioner tolerance to zero and not the outer Krylov preconditioner? Also, if that Krylov solver is being reused, can you confirm that it converged the first time it was called?

bensworth commented 3 years ago

Yep, I set the specific AMG solver tolerance to zero. The error happens before I converge the first time step, so nothing has been reused or previously converged at this point

oseikuffuor1 commented 3 years ago

Ok, can you try calling HYPRE_ClearAllErrors() prior to the call to the solver? If that fixes the issue, then potentially something changed in you problem setup causing hypre to throw the flag (assuming you are using hypre in your setup somehow). If it still fails, then move it to just before calling the setup or solve function for the Krylov solver. If that works, then a closer look might be needed to track what changed in hypre that caused this.

liruipeng commented 3 years ago

@bensworth Can you add --with-print-errors in hypre's build for debugging this? Thanks!

bensworth commented 3 years ago

Well, the HYPRE_ClearAllErrors() did not fix the problem no matter where I put them. But I removed the whole setup of the solver, including options to use an inner HypreGMRES with AMG preconditioning or just AMG preconditioning, block diagonal inverse scaling, etc., and rewrote a simple setup and Mult and it is working. So I don't know where the error was coming from, but the code is working for now. If the error comes back as I add back options I will reopen.