AMReX MLMG bottom solve change causes PeleLMeX EB_PipeFlow test to fail

[baperry2]

After https://github.com/AMReX-Codes/amrex/pull/3991 (stemming from https://github.com/Exawind/amr-wind/issues/886) the PeleLMeX EB_PipeFlow test is failing on the nodal projection on the first timestep: https://github.com/baperry2/PeleLMeX/actions/runs/9784946279/job/27016932023.

Despite the CI failing on this test, I've tried with a few different machines and compilers and haven't been able to reproduce the failure locally. However, turning up verbosity for the nodal projection when running locally, I see that since the change the case takes more iterations for the nodal projection to converge (10 vs 8) and there are more frequent bottom solve failures.

@marchdf @WeiqunZhang @asalmgren @jrood-nrel it looks like you all worked the issue in amr-wind. Any idea what might be happening here in LMeX?

[baperry2]

Here's what I get running locally:

With amrex 36f111b2c (old):

 ====================   NEW TIME STEP   ==================== 
 Est. time step - Conv: 1.208751229e-05, divu: 1e+200
 STEP [0] - Time: 0, dt 1.208751229e-05
   - Advance()::MACProjection()  --> Time: 0.114974
Nodal Projection:
 >> Before projection:
  * On lev 0 max(abs(rhs)) = 739.1881134

MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 3
MLMG: Initial rhs               = 739.1881134
MLMG: Initial residual (resid0) = 739.1881134
MLMG: Bottom solve failed.
MLMG: Iteration   1 Fine resid/bnorm = 0.00144668188
MLMG: Iteration   2 Fine resid/bnorm = 4.887726517e-05
MLMG: Iteration   3 Fine resid/bnorm = 2.942642069e-06
MLMG: Iteration   4 Fine resid/bnorm = 1.881492075e-07
MLMG: Iteration   5 Fine resid/bnorm = 1.168997847e-08
MLMG: Iteration   6 Fine resid/bnorm = 7.832919149e-10
MLMG: Iteration   7 Fine resid/bnorm = 5.112867833e-11
MLMG: Iteration   8 Fine resid/bnorm = 3.276700523e-12
MLMG: Final Iter. 8 resid, resid/bnorm = 2.422098078e-09, 3.276700523e-12
MLMG: Timers: Solve = 0.223404125 Iter = 0.200380083 Bottom = 0.014959123
 >> After projection:
  * On lev 0 max(abs(rhs)) = 125.2587278

   - Advance()::VelocityAdvance  --> Time: 0.469178
 >> PeleLMeX::Advance() --> Time: 0.588156

With amrex 0e3e39b5e (new):

====================   NEW TIME STEP   ==================== 
 Est. time step - Conv: 1.208751229e-05, divu: 1e+200
 STEP [0] - Time: 0, dt 1.208751229e-05
   - Advance()::MACProjection()  --> Time: 0.116781
Nodal Projection:
 >> Before projection:
  * On lev 0 max(abs(rhs)) = 739.1881134

MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 3
MLMG: Initial rhs               = 739.1881134
MLMG: Initial residual (resid0) = 739.1881134
MLMG: Bottom solve failed.
MLMG: Iteration   1 Fine resid/bnorm = 0.001387041191
MLMG: Bottom solve failed.
MLMG: Iteration   2 Fine resid/bnorm = 0.004341203704
MLMG: Bottom solve failed.
MLMG: Iteration   3 Fine resid/bnorm = 9.172997472e-05
MLMG: Bottom solve failed.
MLMG: Iteration   4 Fine resid/bnorm = 0.0001001601963
MLMG: Bottom solve failed.
MLMG: Iteration   5 Fine resid/bnorm = 2.247555741e-06
MLMG: Bottom solve failed.
MLMG: Iteration   6 Fine resid/bnorm = 1.879934597e-07
MLMG: Bottom solve failed.
MLMG: Iteration   7 Fine resid/bnorm = 6.093181729e-09
MLMG: Bottom solve failed.
MLMG: Iteration   8 Fine resid/bnorm = 3.940754528e-10
MLMG: Bottom solve failed.
MLMG: Iteration   9 Fine resid/bnorm = 2.055977701e-11
MLMG: Bottom solve failed.
MLMG: Iteration  10 Fine resid/bnorm = 1.157841855e-12
MLMG: Final Iter. 10 resid, resid/bnorm = 8.558629361e-10, 1.157841855e-12
MLMG: Timers: Solve = 0.316350708 Iter = 0.292605167 Bottom = 0.057702499
 >> After projection:
  * On lev 0 max(abs(rhs)) = 125.2587278

   - Advance()::VelocityAdvance  --> Time: 0.567521
 >> PeleLMeX::Advance() --> Time: 0.688364

[WeiqunZhang]

Could you try https://github.com/AMReX-Codes/amrex/pull/4020?

[WeiqunZhang]

I was able to reproduce the failure with the CI test. And the amrex pr fixes it.

[baperry2]

It looks like this does fix the test that was failing, but I opened a draft PR #397 to make sure all tests pass. Thanks for the quick fix!