AMReX-Fluids / IAMR

A parallel, adaptive mesh refinement (AMR) code that solves the variable-density incompressible Navier-Stokes equations.
https://amrex-fluids.github.io/IAMR/
80 stars 57 forks source link

High density ratios with EB #46

Closed dcoveney closed 3 years ago

dcoveney commented 4 years ago

Hi,

I encounter errors when I try and include EB geometry into a problem with large density ratios (816:1).

The issue seems to be with the nodal projection - it makes the velocity and gradp fields 0, which leads to an inability to obtain a time step estimate:

NavierStokes::advance(): before nodal projection 
max(abs(u/v))  = 0.2998059427  0.2482103605
max(abs(gpx/gpy/p)) = 13270.61685  16865.77065  1158.284095
... Projection::level_project() at level 0
Projection::level_project(): lev: 0, time: 0.142975462
NavierStokes::advance(): after velocity update
max(abs(u/v))  = 0  0
max(abs(gpx/gpy/p)) = 0  0  0

...

NavierStokesBase::estTimeStep() failed to provide a good timestep (probably because initial velocity field is zero with no external forcing).
Use ns.init_dt to provide a reasonable timestep on coarsest level.
Note that ns.init_shrink will be applied to init_dt.
amrex::Abort::0::
 !!!

If I change parameters to do with the projection, e.g. the bottom solver or tolerances, the projection fails outright:

Projection::initialVelocityProject(): iteration 0
After nodal projection:
  lev 0: max(abs(u,v)) = 0 0 
Projection::initialVelocityProject(): time: 0.002017629
done calling initialVelocityProject
calling initialPressureProject
Projection::initialPressureProject(): levels = 0  0
amrex::Abort::0::MLMG failed !!!
SIGABRT

The projection issues seem to occur when the more dense region of fluid interacts with the EB. I do not get this issue when I run with a lower density ratio (I've done it with 10:1 but haven't found the cut-off where it stops working).

I have tried tweaking things such as num_pre_smooth, which works for challenging problems without EB but not when I include geometry.

If there are any suggestions for things to try they would be most welcome. Is this a known limitation of IAMR with EB? I haven't found any examples of EB cases that use larger density ratios.

Thanks!

asalmgren commented 4 years ago

There are several strategies to try -- some in place and some we are actively developing. One of the things that may be affecting your convergence is whether the EB constrains how many multigrid levels you are using.

But I am somewhat confused by your description so let me address it in parts.

First -- you say the projection makes vel and gradp 0 -- is that just that it starts at rest therefore doesn't do anything, or are you saying it starts with a non-divergent field but zeroes the output? The former makes sense, the latter doesn't.

To diagnose convergence issues, it would be helpful to turn on more verbosity for the nodal solve -- could you re-run with nodal_proj.verbose = 4 and nodal_proj.set_verbose = 4?

On Mon, Jun 22, 2020 at 9:35 AM dcoveney notifications@github.com wrote:

Hi,

I encounter errors when I try and include EB geometry into a problem with large density ratios (816:1).

The issue seems to be with the nodal projection - it makes the velocity and gradp fields 0, which leads to an inability to obtain a time step estimate:

NavierStokes::advance(): before nodal projection max(abs(u/v)) = 0.2998059427 0.2482103605 max(abs(gpx/gpy/p)) = 13270.61685 16865.77065 1158.284095 ... Projection::level_project() at level 0 Projection::level_project(): lev: 0, time: 0.142975462 NavierStokes::advance(): after velocity update max(abs(u/v)) = 0 0 max(abs(gpx/gpy/p)) = 0 0 0

...

NavierStokesBase::estTimeStep() failed to provide a good timestep (probably because initial velocity field is zero with no external forcing). Use ns.init_dt to provide a reasonable timestep on coarsest level. Note that ns.init_shrink will be applied to init_dt. amrex::Abort::0:: !!!

If I change parameters to do with the projection, e.g. the bottom solver or tolerances, the projection fails outright:

Projection::initialVelocityProject(): iteration 0 After nodal projection: lev 0: max(abs(u,v)) = 0 0 Projection::initialVelocityProject(): time: 0.002017629 done calling initialVelocityProject calling initialPressureProject Projection::initialPressureProject(): levels = 0 0 amrex::Abort::0::MLMG failed !!! SIGABRT

The projection issues seem to occur when the more dense region of fluid interacts with the EB. I do not get this issue when I run with a lower density ratio (I've done it with 10:1 but haven't found the cut-off where it stops working).

I have tried tweaking things such as num_pre_smooth, which works for challenging problems without EB but not when I include geometry.

If there are any suggestions for things to try they would be most welcome. Is this a known limitation of IAMR with EB? I haven't found any examples of EB cases that use larger density ratios.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/IAMR/issues/46, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YWQWWQ55ZH3YLBN7DLRX6B4DANCNFSM4OE3BKNA .

-- Ann Almgren Senior Scientist; CCSE Group Lead

dcoveney commented 4 years ago

Sorry if I didn't make myself clear; from the terminal output I was getting (the first block above), it appeared that the max values of u and gradp were entering as non-zero values before the nodal projection, then subsequently exiting the projection as zeros, which caused the time step estimate to fail.

I have tried running with verbose, and you are right that it is not converging:

MLMG: Iteration 100 Fine resid/bnorm = 1.30537069e-11
MLMG: Failed to converge after 100 iterations. resid, resid/bnorm = 2.048909835e-08, 1.30537069e-11

Below is the output for an iteration:

MLMG: Iteration  99 Fine resid/bnorm = 1.661380631e-11
MLMG: Subtracting 2.451252856e-10 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 2.632215567e-08
AT LEVEL 0 0   DN: Norm after  smooth 1.533046546e-09
AT LEVEL 0 1   DN: Norm before smooth 1.487990941e-09
AT LEVEL 0 1   DN: Norm after  smooth 8.6165063e-10
AT LEVEL 0 2   DN: Norm before smooth 9.689740517e-10
AT LEVEL 0 2   DN: Norm after  smooth 8.710980452e-10
AT LEVEL 0 3   DN: Norm before smooth 1.0328953e-09
AT LEVEL 0 3   DN: Norm after  smooth 1.047016806e-09
AT LEVEL 0 4   DN: Norm before bottom 8.258735422e-10
MLMG: Subtracting 4.141924451e-23 on level (0, 4)
AT LEVEL 0 4   UP: Norm after  bottom 5.066377749e-14
AT LEVEL 0 3   UP: Norm before smooth 1.866054303e-09
AT LEVEL 0 3   UP: Norm after  smooth 7.081094329e-11
AT LEVEL 0 2   UP: Norm before smooth 8.39176202e-09
AT LEVEL 0 2   UP: Norm after  smooth 1.084439927e-10
AT LEVEL 0 1   UP: Norm before smooth 1.11481313e-08
AT LEVEL 0 1   UP: Norm after  smooth 1.105469029e-10
AT LEVEL 0 0   UP: Norm before smooth 3.560304103e-08
AT LEVEL 0 0   UP: Norm after  smooth 1.195405095e-10

Is this what you mean by the number of multigrid levels being constrained? In each iteration it only works on level(0,0) and level(0,4).

dcoveney commented 4 years ago

Just for further info regarding the projection producing zeros in the u and gradp fields, an example of the output with verbose on is shown below:

MLMG: Iteration   4 Fine resid/bnorm = 7.265789225e+137
MLMG: Subtracting 2.545803384e+128 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 2.280876553e+141
AT LEVEL 0 0   DN: Norm after  smooth 2.245752163e+141
AT LEVEL 0 1   DN: Norm before smooth 1.442198787e+141
AT LEVEL 0 1   DN: Norm after  smooth 2.154750782e+141
AT LEVEL 0 2   DN: Norm before smooth 1.383566624e+141
AT LEVEL 0 2   DN: Norm after  smooth 1.951236035e+141
AT LEVEL 0 3   DN: Norm before smooth 1.252872803e+141
AT LEVEL 0 3   DN: Norm after  smooth 1.999410945e+141
AT LEVEL 0 4   DN: Norm before bottom 1.092194672e+141
MLMG: Subtracting -1.394828094e+125 on level (0, 4)
AT LEVEL 0 4   UP: Norm after  bottom 0
AT LEVEL 0 3   UP: Norm before smooth 0
AT LEVEL 0 3   UP: Norm after  smooth 0
AT LEVEL 0 2   UP: Norm before smooth 0
AT LEVEL 0 2   UP: Norm after  smooth 0
AT LEVEL 0 1   UP: Norm before smooth 0
AT LEVEL 0 1   UP: Norm after  smooth 0
AT LEVEL 0 0   UP: Norm before smooth 0
AT LEVEL 0 0   UP: Norm after  smooth 0
MLMG: Iteration   5 Fine resid/bnorm = 0
MLMG: Final Iter. 5 resid, resid/bnorm = 0, 0
MLMG: Timers: Solve = 0.267126951 Iter = 0.266094068 Bottom = 0.005261424
 >> After projection:
  * On lev 0 max(abs(rhs)) = 0

Projection::level_project(): lev: 0, time: 0.269820623
NavierStokes::advance(): after velocity update
max(abs(u/v))  = 0  0
max(abs(gpx/gpy/p)) = 0  0  0

So the solution blows up with zeros being the result (a trivial solution to the divergence constraint).

asalmgren commented 4 years ago

The zero's are not really the solution being zero, we need a better indicator in the code that the multigrid did not converge and so the simulation should abort.

That said, I'm wondering if you could run your simulation with a multigrid tolerance of 1e-10 for example -- it looks like your multgrid is very close to reaching a good tolerance.

EB + high density ratio is hard for geometric multigrid but it looks like your solvers might be close.

Feel free to contact me directly for more help with this.

Ann

On Tue, Jun 23, 2020 at 7:13 AM dcoveney notifications@github.com wrote:

Just for further info regarding the projection producing zeros in the u and gradp fields, an example of the output with verbose on is shown below:

MLMG: Iteration 4 Fine resid/bnorm = 7.265789225e+137 MLMG: Subtracting 2.545803384e+128 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 2.280876553e+141 AT LEVEL 0 0 DN: Norm after smooth 2.245752163e+141 AT LEVEL 0 1 DN: Norm before smooth 1.442198787e+141 AT LEVEL 0 1 DN: Norm after smooth 2.154750782e+141 AT LEVEL 0 2 DN: Norm before smooth 1.383566624e+141 AT LEVEL 0 2 DN: Norm after smooth 1.951236035e+141 AT LEVEL 0 3 DN: Norm before smooth 1.252872803e+141 AT LEVEL 0 3 DN: Norm after smooth 1.999410945e+141 AT LEVEL 0 4 DN: Norm before bottom 1.092194672e+141 MLMG: Subtracting -1.394828094e+125 on level (0, 4) AT LEVEL 0 4 UP: Norm after bottom 0 AT LEVEL 0 3 UP: Norm before smooth 0 AT LEVEL 0 3 UP: Norm after smooth 0 AT LEVEL 0 2 UP: Norm before smooth 0 AT LEVEL 0 2 UP: Norm after smooth 0 AT LEVEL 0 1 UP: Norm before smooth 0 AT LEVEL 0 1 UP: Norm after smooth 0 AT LEVEL 0 0 UP: Norm before smooth 0 AT LEVEL 0 0 UP: Norm after smooth 0 MLMG: Iteration 5 Fine resid/bnorm = 0 MLMG: Final Iter. 5 resid, resid/bnorm = 0, 0 MLMG: Timers: Solve = 0.267126951 Iter = 0.266094068 Bottom = 0.005261424

After projection:

  • On lev 0 max(abs(rhs)) = 0

Projection::level_project(): lev: 0, time: 0.269820623 NavierStokes::advance(): after velocity update max(abs(u/v)) = 0 0 max(abs(gpx/gpy/p)) = 0 0 0

So the solution blows up with zeros being the result (a trivial solution to the divergence constraint).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/IAMR/issues/46#issuecomment-648177504, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YS3GFQ4OQI65T3P2WTRYC2CDANCNFSM4OE3BKNA .

-- Ann Almgren Senior Scientist; CCSE Group Lead

dcoveney commented 4 years ago

Hi Ann,

Thanks for your response - can you confirm that I would set the MG tolerance in this case with proj.proj_tol? I deduced this from the PP query in Projection.cpp that assigns the rel_tol for the
nodal_projector.project(phi_rebase,rel_tol,abs_tol); call, but then other aspects of the nodal projection are preceded by nodal_proj.

asalmgren commented 4 years ago

Yes, use

proj.proj_tol

It is confusing -- a lot of the behavior of the projection is controlled using nodal_proj.* but the tolerance is passed in the "project" call..

On Wed, Jul 1, 2020 at 10:42 AM dcoveney notifications@github.com wrote:

Hi Ann,

Thanks for your response - can you confirm that I would set the MG tolerance in this case with proj.proj_tol? I deduced this from the PP query in Projection.cpp that assigns the rel_tol for the nodal_projector.project(phi_rebase,rel_tol,abs_tol); call, but then other aspects of the nodal projection are preceded by nodal_proj.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/IAMR/issues/46#issuecomment-652557723, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YTPGOAPWOUVB2SWKODRZNYRPANCNFSM4OE3BKNA .

-- Ann Almgren Senior Scientist; CCSE Group Lead

dcoveney commented 4 years ago

Thanks for this.

I just tried this with proj.proj_tol = 1.0e-10 and I had a similar crash. Here is the terminal output for the last two time steps:

[STEP 4] Coarse TimeStep time: 1.709251631
[STEP 4] FAB kilobyte spread across MPI nodes: [226900 ... 226900]

STEP = 4 TIME = 0.0002639936952 DT = 7.571118472e-05

PLOTFILE: file = plt00004
Write plotfile time = 0.057479739  seconds

estTimeStep :: 
LEV = 0 UMAX = 1.918219757  1.880784946  
estimated timestep: dt = 0.0002036393373
[Level 0 step 5] ADVANCE with dt = 8.328230319e-05
Advancing grids at level 0 : starting time = 0.0002639936952 with dt = 8.328230319e-05
NavierStokes::advance(): before velocity update:
max(abs(u/v))  = 1.918219757  1.880784946
max(abs(gpx/gpy/p)) = 56047.27994  12870.78883  4662.571817
... predict edge velocities
... mac_projection
... mac_project at level 0
SUM of DIV(U_edge) = -6656
NavierStokesBase:mac_project(): lev: 0, time: 0.193923521
... advect velocities
... advect scalars
... update scalars
... update scalars
... update velocities 
NavierStokes:velocity_diffusion_update(): lev: 0, time: 1.830003384e-07
NavierStokes::advance(): before nodal projection 
max(abs(u/v))  = 1.92277004  1.834439165
max(abs(gpx/gpy/p)) = 56047.27994  12870.78883  4671.796138
... Projection::level_project() at level 0
Nodal Projection:
 >> Before projection:
  * On lev 0 max(abs(rhs)) = 726976.8117

MLMG: Subtracting -609.7483866 from rhs
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 7
MLMG: Initial rhs               = 727586.5601
MLMG: Initial residual (resid0) = 727586.5601
MLMG: Subtracting 4.303918333e-13 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 727586.5601
AT LEVEL 0 0   DN: Norm after  smooth 5781.929765
AT LEVEL 0 1   DN: Norm before smooth 3885.712465
AT LEVEL 0 1   DN: Norm after  smooth 1551.668289
AT LEVEL 0 2   DN: Norm before smooth 1053.841467
AT LEVEL 0 2   DN: Norm after  smooth 667.2690317
AT LEVEL 0 3   DN: Norm before smooth 435.7146835
AT LEVEL 0 3   DN: Norm after  smooth 125.2013589
AT LEVEL 0 4   DN: Norm before smooth 82.91739746
AT LEVEL 0 4   DN: Norm after  smooth 51.5325694
AT LEVEL 0 5   DN: Norm before smooth 34.58015972
AT LEVEL 0 5   DN: Norm after  smooth 4.304889328
AT LEVEL 0 6   DN: Norm before bottom 2.774173854
MLMG: Subtracting 2.368244489e-14 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 0.04542755851
AT LEVEL 0 5   UP: Norm before smooth 6.071179849
AT LEVEL 0 5   UP: Norm after  smooth 0.04217304486
AT LEVEL 0 4   UP: Norm before smooth 46.36686106
AT LEVEL 0 4   UP: Norm after  smooth 1.943684422
AT LEVEL 0 3   UP: Norm before smooth 139.0154107
AT LEVEL 0 3   UP: Norm after  smooth 7.230719772
AT LEVEL 0 2   UP: Norm before smooth 797.1871627
AT LEVEL 0 2   UP: Norm after  smooth 22.78696314
AT LEVEL 0 1   UP: Norm before smooth 4421.163206
AT LEVEL 0 1   UP: Norm after  smooth 68.45611824
AT LEVEL 0 0   UP: Norm before smooth 15103.67505
AT LEVEL 0 0   UP: Norm after  smooth 308.9836274
MLMG: Iteration   1 Fine resid/bnorm = 0.0004246692344
MLMG: Subtracting 1.390484949e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 308.9836274
AT LEVEL 0 0   DN: Norm after  smooth 514.7721261
AT LEVEL 0 1   DN: Norm before smooth 199.2470902
AT LEVEL 0 1   DN: Norm after  smooth 50.36863492
AT LEVEL 0 2   DN: Norm before smooth 34.71612638
AT LEVEL 0 2   DN: Norm after  smooth 26.71743829
AT LEVEL 0 3   DN: Norm before smooth 17.43863803
AT LEVEL 0 3   DN: Norm after  smooth 8.102352416
AT LEVEL 0 4   DN: Norm before smooth 5.507201959
AT LEVEL 0 4   DN: Norm after  smooth 2.08050573
AT LEVEL 0 5   DN: Norm before smooth 1.330782877
AT LEVEL 0 5   DN: Norm after  smooth 0.1863928778
AT LEVEL 0 6   DN: Norm before bottom 0.1244828817
MLMG: Subtracting -3.638582491e-16 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 0.0007790313336
AT LEVEL 0 5   UP: Norm before smooth 0.1648968546
AT LEVEL 0 5   UP: Norm after  smooth 0.0005600761544
AT LEVEL 0 4   UP: Norm before smooth 1.733872978
AT LEVEL 0 4   UP: Norm after  smooth 0.07434496825
AT LEVEL 0 3   UP: Norm before smooth 7.641405832
AT LEVEL 0 3   UP: Norm after  smooth 0.3429656375
AT LEVEL 0 2   UP: Norm before smooth 30.35014544
AT LEVEL 0 2   UP: Norm after  smooth 1.148433919
AT LEVEL 0 1   UP: Norm before smooth 174.7750096
AT LEVEL 0 1   UP: Norm after  smooth 2.047428595
AT LEVEL 0 0   UP: Norm before smooth 705.4816146
AT LEVEL 0 0   UP: Norm after  smooth 89.52163722
MLMG: Iteration   2 Fine resid/bnorm = 0.0001230391573
MLMG: Subtracting 1.23019997e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 89.52163722
AT LEVEL 0 0   DN: Norm after  smooth 96.25353282
AT LEVEL 0 1   DN: Norm before smooth 37.01451762
AT LEVEL 0 1   DN: Norm after  smooth 2.861900667
AT LEVEL 0 2   DN: Norm before smooth 1.574315764
AT LEVEL 0 2   DN: Norm after  smooth 1.168929959
AT LEVEL 0 3   DN: Norm before smooth 0.7623740571
AT LEVEL 0 3   DN: Norm after  smooth 0.2727603352
AT LEVEL 0 4   DN: Norm before smooth 0.1816480787
AT LEVEL 0 4   DN: Norm after  smooth 0.1009015168
AT LEVEL 0 5   DN: Norm before smooth 0.06322336384
AT LEVEL 0 5   DN: Norm after  smooth 0.009795415901
AT LEVEL 0 6   DN: Norm before bottom 0.006116800093
MLMG: Subtracting -7.155734338e-18 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 0.0002367386316
AT LEVEL 0 5   UP: Norm before smooth 0.008324693203
AT LEVEL 0 5   UP: Norm after  smooth 0.0001496849684
AT LEVEL 0 4   UP: Norm before smooth 0.08471655633
AT LEVEL 0 4   UP: Norm after  smooth 0.003627525886
AT LEVEL 0 3   UP: Norm before smooth 0.3366393005
AT LEVEL 0 3   UP: Norm after  smooth 0.01577659517
AT LEVEL 0 2   UP: Norm before smooth 1.334980616
AT LEVEL 0 2   UP: Norm after  smooth 0.0569644283
AT LEVEL 0 1   UP: Norm before smooth 7.458849476
AT LEVEL 0 1   UP: Norm after  smooth 0.110163214
AT LEVEL 0 0   UP: Norm before smooth 44.52699824
AT LEVEL 0 0   UP: Norm after  smooth 13.93153677
MLMG: Iteration   3 Fine resid/bnorm = 1.914760049e-05
MLMG: Subtracting 1.212006027e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 13.93153678
AT LEVEL 0 0   DN: Norm after  smooth 13.7102312
AT LEVEL 0 1   DN: Norm before smooth 5.263212122
AT LEVEL 0 1   DN: Norm after  smooth 0.2208144174
AT LEVEL 0 2   DN: Norm before smooth 0.1069266951
AT LEVEL 0 2   DN: Norm after  smooth 0.05662250156
AT LEVEL 0 3   DN: Norm before smooth 0.03523740058
AT LEVEL 0 3   DN: Norm after  smooth 0.01843526223
AT LEVEL 0 4   DN: Norm before smooth 0.01258629828
AT LEVEL 0 4   DN: Norm after  smooth 0.005108690705
AT LEVEL 0 5   DN: Norm before smooth 0.003352680777
AT LEVEL 0 5   DN: Norm after  smooth 0.0004701936047
AT LEVEL 0 6   DN: Norm before bottom 0.0003149877577
MLMG: Subtracting -3.904821887e-19 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 1.027752768e-05
AT LEVEL 0 5   UP: Norm before smooth 0.0004120262623
AT LEVEL 0 5   UP: Norm after  smooth 6.043355783e-06
AT LEVEL 0 4   UP: Norm before smooth 0.004386870411
AT LEVEL 0 4   UP: Norm after  smooth 0.0001879987059
AT LEVEL 0 3   UP: Norm before smooth 0.0187712218
AT LEVEL 0 3   UP: Norm after  smooth 0.0008508209144
AT LEVEL 0 2   UP: Norm before smooth 0.07193143351
AT LEVEL 0 2   UP: Norm after  smooth 0.003124074591
AT LEVEL 0 1   UP: Norm before smooth 0.3392636185
AT LEVEL 0 1   UP: Norm after  smooth 0.006303778595
AT LEVEL 0 0   UP: Norm before smooth 6.215739699
AT LEVEL 0 0   UP: Norm after  smooth 1.887033965
MLMG: Iteration   4 Fine resid/bnorm = 2.593552534e-06
MLMG: Subtracting 1.376882928e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 1.887033967
AT LEVEL 0 0   DN: Norm after  smooth 1.8073092
AT LEVEL 0 1   DN: Norm before smooth 0.6934140883
AT LEVEL 0 1   DN: Norm after  smooth 0.02103211361
AT LEVEL 0 2   DN: Norm before smooth 0.01002116655
AT LEVEL 0 2   DN: Norm after  smooth 0.003557997337
AT LEVEL 0 3   DN: Norm before smooth 0.002077650965
AT LEVEL 0 3   DN: Norm after  smooth 0.000736779751
AT LEVEL 0 4   DN: Norm before smooth 0.0004711199177
AT LEVEL 0 4   DN: Norm after  smooth 0.0002628469375
AT LEVEL 0 5   DN: Norm before smooth 0.0001686099859
AT LEVEL 0 5   DN: Norm after  smooth 2.814042791e-05
AT LEVEL 0 6   DN: Norm before bottom 1.760594174e-05
MLMG: Subtracting 1.196434038e-20 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 1.245030424e-06
AT LEVEL 0 5   UP: Norm before smooth 2.131731721e-05
AT LEVEL 0 5   UP: Norm after  smooth 8.670749792e-07
AT LEVEL 0 4   UP: Norm before smooth 0.0002241844423
AT LEVEL 0 4   UP: Norm after  smooth 9.600649717e-06
AT LEVEL 0 3   UP: Norm before smooth 0.0009122373209
AT LEVEL 0 3   UP: Norm after  smooth 4.21707361e-05
AT LEVEL 0 2   UP: Norm before smooth 0.003935915679
AT LEVEL 0 2   UP: Norm after  smooth 0.0001599380214
AT LEVEL 0 1   UP: Norm before smooth 0.01504381927
AT LEVEL 0 1   UP: Norm after  smooth 0.0003978510211
AT LEVEL 0 0   UP: Norm before smooth 0.814837668
AT LEVEL 0 0   UP: Norm after  smooth 0.2438598809
MLMG: Iteration   5 Fine resid/bnorm = 3.351627087e-07
MLMG: Subtracting 1.362447237e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 0.2438598823
AT LEVEL 0 0   DN: Norm after  smooth 0.230853851
AT LEVEL 0 1   DN: Norm before smooth 0.08855015765
AT LEVEL 0 1   DN: Norm after  smooth 0.002231679449
AT LEVEL 0 2   DN: Norm before smooth 0.001050100473
AT LEVEL 0 2   DN: Norm after  smooth 0.0002306821717
AT LEVEL 0 3   DN: Norm before smooth 0.0001327772516
AT LEVEL 0 3   DN: Norm after  smooth 4.575058407e-05
AT LEVEL 0 4   DN: Norm before smooth 3.13573135e-05
AT LEVEL 0 4   DN: Norm after  smooth 1.452837284e-05
AT LEVEL 0 5   DN: Norm before smooth 9.087575767e-06
AT LEVEL 0 5   DN: Norm after  smooth 1.457197901e-06
AT LEVEL 0 6   DN: Norm before bottom 9.800742062e-07
MLMG: Subtracting -3.67268192e-21 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 8.063609705e-08
AT LEVEL 0 5   UP: Norm before smooth 1.089846713e-06
AT LEVEL 0 5   UP: Norm after  smooth 5.223839905e-08
AT LEVEL 0 4   UP: Norm before smooth 1.187499724e-05
AT LEVEL 0 4   UP: Norm after  smooth 5.087963902e-07
AT LEVEL 0 3   UP: Norm before smooth 5.100428209e-05
AT LEVEL 0 3   UP: Norm after  smooth 2.251302182e-06
AT LEVEL 0 2   UP: Norm before smooth 0.0002161870219
AT LEVEL 0 2   UP: Norm after  smooth 8.56632633e-06
AT LEVEL 0 1   UP: Norm before smooth 0.0007693354972
AT LEVEL 0 1   UP: Norm after  smooth 2.648630204e-05
AT LEVEL 0 0   UP: Norm before smooth 0.1037347356
AT LEVEL 0 0   UP: Norm after  smooth 0.03090814427
MLMG: Iteration   6 Fine resid/bnorm = 4.248037238e-08
MLMG: Subtracting 1.200004533e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 0.03090814801
AT LEVEL 0 0   DN: Norm after  smooth 0.02912779376
AT LEVEL 0 1   DN: Norm before smooth 0.01117165159
AT LEVEL 0 1   DN: Norm after  smooth 0.0002590554468
AT LEVEL 0 2   DN: Norm before smooth 0.0001210958103
AT LEVEL 0 2   DN: Norm after  smooth 1.839192302e-05
AT LEVEL 0 3   DN: Norm before smooth 1.037546348e-05
AT LEVEL 0 3   DN: Norm after  smooth 2.083599409e-06
AT LEVEL 0 4   DN: Norm before smooth 1.334406961e-06
AT LEVEL 0 4   DN: Norm after  smooth 7.115565523e-07
AT LEVEL 0 5   DN: Norm before smooth 4.699012154e-07
AT LEVEL 0 5   DN: Norm after  smooth 9.256460306e-08
AT LEVEL 0 6   DN: Norm before bottom 5.808612107e-08
MLMG: Subtracting 4.063524759e-22 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 7.236287323e-09
AT LEVEL 0 5   UP: Norm before smooth 5.749241134e-08
AT LEVEL 0 5   UP: Norm after  smooth 5.078890277e-09
AT LEVEL 0 4   UP: Norm before smooth 6.215994981e-07
AT LEVEL 0 4   UP: Norm after  smooth 2.662380058e-08
AT LEVEL 0 3   UP: Norm before smooth 2.569960313e-06
AT LEVEL 0 3   UP: Norm after  smooth 1.17776129e-07
AT LEVEL 0 2   UP: Norm before smooth 1.18413588e-05
AT LEVEL 0 2   UP: Norm after  smooth 4.347492031e-07
AT LEVEL 0 1   UP: Norm before smooth 8.915226982e-05
AT LEVEL 0 1   UP: Norm after  smooth 1.760088333e-06
AT LEVEL 0 0   UP: Norm before smooth 0.01307474151
AT LEVEL 0 0   UP: Norm after  smooth 0.003886744124
MLMG: Iteration   7 Fine resid/bnorm = 5.341969347e-09
MLMG: Subtracting 1.21705513e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 0.003886745103
AT LEVEL 0 0   DN: Norm after  smooth 0.003655464318
AT LEVEL 0 1   DN: Norm before smooth 0.001401952173
AT LEVEL 0 1   DN: Norm after  smooth 3.123344065e-05
AT LEVEL 0 2   DN: Norm before smooth 1.45491972e-05
AT LEVEL 0 2   DN: Norm after  smooth 1.698448809e-06
AT LEVEL 0 3   DN: Norm before smooth 9.418170222e-07
AT LEVEL 0 3   DN: Norm after  smooth 1.418625994e-07
AT LEVEL 0 4   DN: Norm before smooth 8.70467465e-08
AT LEVEL 0 4   DN: Norm after  smooth 4.485896078e-08
AT LEVEL 0 5   DN: Norm before smooth 2.738667329e-08
AT LEVEL 0 5   DN: Norm after  smooth 5.385651795e-09
AT LEVEL 0 6   DN: Norm before bottom 3.642732964e-09
MLMG: Subtracting 4.34592939e-24 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 5.639027261e-10
AT LEVEL 0 5   UP: Norm before smooth 2.999543576e-09
AT LEVEL 0 5   UP: Norm after  smooth 3.753608052e-10
AT LEVEL 0 4   UP: Norm before smooth 3.335717624e-08
AT LEVEL 0 4   UP: Norm after  smooth 1.428924253e-09
AT LEVEL 0 3   UP: Norm before smooth 1.434681762e-07
AT LEVEL 0 3   UP: Norm after  smooth 6.512329949e-09
AT LEVEL 0 2   UP: Norm before smooth 6.508552765e-07
AT LEVEL 0 2   UP: Norm after  smooth 2.255522747e-08
AT LEVEL 0 1   UP: Norm before smooth 1.070303661e-05
AT LEVEL 0 1   UP: Norm after  smooth 1.190010584e-07
AT LEVEL 0 0   UP: Norm before smooth 0.001639860996
AT LEVEL 0 0   UP: Norm after  smooth 0.0004870777249
MLMG: Iteration   8 Fine resid/bnorm = 6.694449144e-10
MLMG: Subtracting 1.296554173e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 0.0004870791211
AT LEVEL 0 0   DN: Norm after  smooth 0.0004577057848
AT LEVEL 0 1   DN: Norm before smooth 0.0001755371309
AT LEVEL 0 1   DN: Norm after  smooth 3.843425417e-06
AT LEVEL 0 2   DN: Norm before smooth 1.787593864e-06
AT LEVEL 0 2   DN: Norm after  smooth 1.809471265e-07
AT LEVEL 0 3   DN: Norm before smooth 9.904325601e-08
AT LEVEL 0 3   DN: Norm after  smooth 1.153612554e-08
AT LEVEL 0 4   DN: Norm before smooth 6.645044494e-09
AT LEVEL 0 4   DN: Norm after  smooth 2.17561113e-09
AT LEVEL 0 5   DN: Norm before smooth 1.386879828e-09
AT LEVEL 0 5   DN: Norm after  smooth 3.550225092e-10
AT LEVEL 0 6   DN: Norm before bottom 2.235719467e-10
MLMG: Subtracting 5.10687093e-24 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 4.28276965e-11
AT LEVEL 0 5   UP: Norm before smooth 1.601541119e-10
AT LEVEL 0 5   UP: Norm after  smooth 2.962254078e-11
AT LEVEL 0 4   UP: Norm before smooth 1.773505625e-09
AT LEVEL 0 4   UP: Norm after  smooth 7.597946366e-11
AT LEVEL 0 3   UP: Norm before smooth 7.416892851e-09
AT LEVEL 0 3   UP: Norm after  smooth 3.299500371e-10
AT LEVEL 0 2   UP: Norm before smooth 6.044005136e-08
AT LEVEL 0 2   UP: Norm after  smooth 1.466266586e-09
AT LEVEL 0 1   UP: Norm before smooth 1.316416734e-06
AT LEVEL 0 1   UP: Norm after  smooth 8.236023845e-09
AT LEVEL 0 0   UP: Norm before smooth 0.0002052859254
AT LEVEL 0 0   UP: Norm after  smooth 6.095071136e-05
MLMG: Iteration   9 Fine resid/bnorm = 8.377293097e-11
MLMG: Final Iter. 9 resid, resid/bnorm = 6.095205867e-05, 8.377293097e-11
MLMG: Timers: Solve = 1.756000754 Iter = 1.746957118 Bottom = 0.003314982
 >> After projection:
  * On lev 0 max(abs(rhs)) = 167063.8425

Projection::level_project(): lev: 0, time: 1.778533854
NavierStokes::advance(): after velocity update
max(abs(u/v))  = 1.92228911  1.800679662
max(abs(gpx/gpy/p)) = 49053.33771  13697.83594  4658.531877
[Level 0 step 5] Advanced 131072 cells

[STEP 5] Coarse TimeStep time: 2.167339015
[STEP 5] FAB kilobyte spread across MPI nodes: [226900 ... 226900]

STEP = 5 TIME = 0.0003472759984 DT = 8.328230319e-05

PLOTFILE: file = plt00005
Write plotfile time = 0.056482354  seconds

estTimeStep :: 
LEV = 0 UMAX = 1.92228911  1.800679662  
estimated timestep: dt = 0.0002032082468
[Level 0 step 6] ADVANCE with dt = 9.161053351e-05
Advancing grids at level 0 : starting time = 0.0003472759984 with dt = 9.161053351e-05
NavierStokes::advance(): before velocity update:
max(abs(u/v))  = 1.92228911  1.800679662
max(abs(gpx/gpy/p)) = 49053.33771  13697.83594  4658.531877
... predict edge velocities
... mac_projection
... mac_project at level 0
SUM of DIV(U_edge) = -6656.00000000001
NavierStokesBase:mac_project(): lev: 0, time: 0.193445177
... advect velocities
... advect scalars
... update scalars
... update scalars
... update velocities 
NavierStokes:velocity_diffusion_update(): lev: 0, time: 2.229999154e-07
NavierStokes::advance(): before nodal projection 
max(abs(u/v))  = 1.927049302  1.745457294
max(abs(gpx/gpy/p)) = 49053.33771  13697.83594  4662.571817
... Projection::level_project() at level 0
Nodal Projection:
 >> Before projection:
  * On lev 0 max(abs(rhs)) = 700906.1935

MLMG: Subtracting -554.3167151 from rhs
MLMG: # of AMR levels: 1
      # of MG levels on the coarsest AMR level: 7
MLMG: Initial rhs               = 701460.5102
MLMG: Initial residual (resid0) = 701460.5102
MLMG: Subtracting -1.641131675e-12 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 701460.5102
AT LEVEL 0 0   DN: Norm after  smooth 5486875205
AT LEVEL 0 1   DN: Norm before smooth 1757307913
AT LEVEL 0 1   DN: Norm after  smooth 40845685.58
AT LEVEL 0 2   DN: Norm before smooth 16490086.69
AT LEVEL 0 2   DN: Norm after  smooth 380767.8523
AT LEVEL 0 3   DN: Norm before smooth 172970.1855
AT LEVEL 0 3   DN: Norm after  smooth 15917.43893
AT LEVEL 0 4   DN: Norm before smooth 8716.572349
AT LEVEL 0 4   DN: Norm after  smooth 773.881569
AT LEVEL 0 5   DN: Norm before smooth 433.3378402
AT LEVEL 0 5   DN: Norm after  smooth 51.27691564
AT LEVEL 0 6   DN: Norm before bottom 34.7657818
MLMG: Subtracting 1.203807887e-11 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 8.042035863
AT LEVEL 0 5   UP: Norm before smooth 32.25063201
AT LEVEL 0 5   UP: Norm after  smooth 5.32865551
AT LEVEL 0 4   UP: Norm before smooth 270.7536296
AT LEVEL 0 4   UP: Norm after  smooth 7.061239781
AT LEVEL 0 3   UP: Norm before smooth 3549.853286
AT LEVEL 0 3   UP: Norm after  smooth 37.22150095
AT LEVEL 0 2   UP: Norm before smooth 118371.245
AT LEVEL 0 2   UP: Norm after  smooth 1137.091865
AT LEVEL 0 1   UP: Norm before smooth 17575399.23
AT LEVEL 0 1   UP: Norm after  smooth 1106075.05
AT LEVEL 0 0   UP: Norm before smooth 2410577609
AT LEVEL 0 0   UP: Norm after  smooth 4.840725006e+16
MLMG: Iteration   1 Fine resid/bnorm = 6.900923054e+10
MLMG: Subtracting 9.433302616e-05 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 4.840725006e+16
AT LEVEL 0 0   DN: Norm after  smooth 6.436612934e+23
AT LEVEL 0 1   DN: Norm before smooth 2.061484984e+23
AT LEVEL 0 1   DN: Norm after  smooth 4.791580227e+21
AT LEVEL 0 2   DN: Norm before smooth 1.934441183e+21
AT LEVEL 0 2   DN: Norm after  smooth 4.466958135e+19
AT LEVEL 0 3   DN: Norm before smooth 2.029213339e+19
AT LEVEL 0 3   DN: Norm after  smooth 1.869435319e+18
AT LEVEL 0 4   DN: Norm before smooth 1.023794371e+18
AT LEVEL 0 4   DN: Norm after  smooth 9.065508364e+16
AT LEVEL 0 5   DN: Norm before smooth 5.010658131e+16
AT LEVEL 0 5   DN: Norm after  smooth 5.600964301e+15
AT LEVEL 0 6   DN: Norm before bottom 3.798039069e+15
MLMG: Subtracting 143.8515625 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 9.460990068e+14
AT LEVEL 0 5   UP: Norm before smooth 3.801645159e+15
AT LEVEL 0 5   UP: Norm after  smooth 6.25917822e+14
AT LEVEL 0 4   UP: Norm before smooth 3.168932189e+16
AT LEVEL 0 4   UP: Norm after  smooth 9.936955581e+14
AT LEVEL 0 3   UP: Norm before smooth 4.247572335e+17
AT LEVEL 0 3   UP: Norm after  smooth 5.306523363e+15
AT LEVEL 0 2   UP: Norm before smooth 1.389636299e+19
AT LEVEL 0 2   UP: Norm after  smooth 1.365857551e+17
AT LEVEL 0 1   UP: Norm before smooth 2.061815135e+21
AT LEVEL 0 1   UP: Norm after  smooth 1.297384974e+20
AT LEVEL 0 0   UP: Norm before smooth 2.827831961e+23
AT LEVEL 0 0   UP: Norm after  smooth 5.678614925e+30
MLMG: Iteration   2 Fine resid/bnorm = 8.09541641e+24
MLMG: Subtracting -1.157167652e+10 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 5.678614925e+30
AT LEVEL 0 0   DN: Norm after  smooth 7.550738004e+37
AT LEVEL 0 1   DN: Norm before smooth 2.41831118e+37
AT LEVEL 0 1   DN: Norm after  smooth 5.620963587e+35
AT LEVEL 0 2   DN: Norm before smooth 2.269277135e+35
AT LEVEL 0 2   DN: Norm after  smooth 5.240152065e+33
AT LEVEL 0 3   DN: Norm before smooth 2.380453576e+33
AT LEVEL 0 3   DN: Norm after  smooth 2.193019288e+32
AT LEVEL 0 4   DN: Norm before smooth 1.201004805e+32
AT LEVEL 0 4   DN: Norm after  smooth 1.063467374e+31
AT LEVEL 0 5   DN: Norm before smooth 5.877962084e+30
AT LEVEL 0 5   DN: Norm after  smooth 6.570445426e+29
AT LEVEL 0 6   DN: Norm before bottom 4.455448578e+29
MLMG: Subtracting 2.814077691e+17 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 1.109861009e+29
AT LEVEL 0 5   UP: Norm before smooth 4.459678852e+29
AT LEVEL 0 5   UP: Norm after  smooth 7.342590793e+28
AT LEVEL 0 4   UP: Norm before smooth 3.717448441e+30
AT LEVEL 0 4   UP: Norm after  smooth 1.165696134e+29
AT LEVEL 0 3   UP: Norm before smooth 4.982792377e+31
AT LEVEL 0 3   UP: Norm after  smooth 6.22503917e+29
AT LEVEL 0 2   UP: Norm before smooth 1.630170981e+33
AT LEVEL 0 2   UP: Norm after  smooth 1.602276326e+31
AT LEVEL 0 1   UP: Norm before smooth 2.418698477e+35
AT LEVEL 0 1   UP: Norm after  smooth 1.521951705e+34
AT LEVEL 0 0   UP: Norm before smooth 3.317306552e+37
AT LEVEL 0 0   UP: Norm after  smooth 6.661536738e+44
MLMG: Iteration   3 Fine resid/bnorm = 9.496666799e+38
MLMG: Subtracting -1.682596856e+24 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 6.661536738e+44
AT LEVEL 0 0   DN: Norm after  smooth 8.857709016e+51
AT LEVEL 0 1   DN: Norm before smooth 2.836901072e+51
AT LEVEL 0 1   DN: Norm after  smooth 6.593906425e+49
AT LEVEL 0 2   DN: Norm before smooth 2.662070453e+49
AT LEVEL 0 2   DN: Norm after  smooth 6.147179542e+47
AT LEVEL 0 3   DN: Norm before smooth 2.792490626e+47
AT LEVEL 0 3   DN: Norm after  smooth 2.572612996e+46
AT LEVEL 0 4   DN: Norm before smooth 1.408888917e+46
AT LEVEL 0 4   DN: Norm after  smooth 1.247544881e+45
AT LEVEL 0 5   DN: Norm before smooth 6.89538926e+44
AT LEVEL 0 5   DN: Norm after  smooth 7.707735807e+43
AT LEVEL 0 6   DN: Norm before bottom 5.226650298e+43
MLMG: Subtracting 2.294850691e+31 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 1.301968875e+43
AT LEVEL 0 5   UP: Norm before smooth 5.231612798e+43
AT LEVEL 0 5   UP: Norm after  smooth 8.613533224e+42
AT LEVEL 0 4   UP: Norm before smooth 4.360908371e+44
AT LEVEL 0 4   UP: Norm after  smooth 1.367468604e+43
AT LEVEL 0 3   UP: Norm before smooth 5.845272996e+45
AT LEVEL 0 3   UP: Norm after  smooth 7.302542552e+43
AT LEVEL 0 2   UP: Norm before smooth 1.912340249e+47
AT LEVEL 0 2   UP: Norm after  smooth 1.879617257e+45
AT LEVEL 0 1   UP: Norm before smooth 2.837355407e+49
AT LEVEL 0 1   UP: Norm after  smooth 1.785389101e+48
AT LEVEL 0 0   UP: Norm before smooth 3.891505193e+51
AT LEVEL 0 0   UP: Norm after  smooth 7.81459428e+58
MLMG: Iteration   4 Fine resid/bnorm = 1.114046217e+53
MLMG: Subtracting 6.717107368e+37 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 7.81459428e+58
AT LEVEL 0 0   DN: Norm after  smooth 1.039090602e+66
AT LEVEL 0 1   DN: Norm before smooth 3.327945452e+65
AT LEVEL 0 1   DN: Norm after  smooth 7.735257713e+63
AT LEVEL 0 2   DN: Norm before smooth 3.122853082e+63
AT LEVEL 0 2   DN: Norm after  smooth 7.211206059e+61
AT LEVEL 0 3   DN: Norm before smooth 3.275847921e+61
AT LEVEL 0 3   DN: Norm after  smooth 3.017911271e+60
AT LEVEL 0 4   DN: Norm before smooth 1.652756069e+60
AT LEVEL 0 4   DN: Norm after  smooth 1.4634847e+59
AT LEVEL 0 5   DN: Norm before smooth 8.088924762e+58
AT LEVEL 0 5   DN: Norm after  smooth 9.041881854e+57
AT LEVEL 0 6   DN: Norm before bottom 6.131340729e+57
MLMG: Subtracting -2.434216717e+44 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 1.527329042e+57
AT LEVEL 0 5   UP: Norm before smooth 6.137162198e+57
AT LEVEL 0 5   UP: Norm after  smooth 1.01044654e+57
AT LEVEL 0 4   UP: Norm before smooth 5.115745954e+58
AT LEVEL 0 4   UP: Norm after  smooth 1.604166239e+57
AT LEVEL 0 3   UP: Norm before smooth 6.857041959e+59
AT LEVEL 0 3   UP: Norm after  smooth 8.566552959e+57
AT LEVEL 0 2   UP: Norm before smooth 2.24335071e+61
AT LEVEL 0 2   UP: Norm after  smooth 2.204963636e+59
AT LEVEL 0 1   UP: Norm before smooth 3.328478428e+63
AT LEVEL 0 1   UP: Norm after  smooth 2.094425357e+62
AT LEVEL 0 0   UP: Norm before smooth 4.565092923e+65
AT LEVEL 0 0   UP: Norm after  smooth 9.167236655e+72
MLMG: Iteration   5 Fine resid/bnorm = 1.306878509e+67
MLMG: Subtracting -8.094118466e+51 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 9.167236655e+72
AT LEVEL 0 0   DN: Norm after  smooth 1.21894869e+80
AT LEVEL 0 1   DN: Norm before smooth 3.903985599e+79
AT LEVEL 0 1   DN: Norm after  smooth 9.074167577e+77
AT LEVEL 0 2   DN: Norm before smooth 3.663393417e+77
AT LEVEL 0 2   DN: Norm after  smooth 8.459406867e+75
AT LEVEL 0 3   DN: Norm before smooth 3.842870412e+75
AT LEVEL 0 3   DN: Norm after  smooth 3.540287036e+74
AT LEVEL 0 4   DN: Norm before smooth 1.938834631e+74
AT LEVEL 0 4   DN: Norm after  smooth 1.716801935e+73
AT LEVEL 0 5   DN: Norm before smooth 9.489051501e+72
AT LEVEL 0 5   DN: Norm after  smooth 1.060695767e+72
AT LEVEL 0 6   DN: Norm before bottom 7.192625678e+71
MLMG: Subtracting -8.30658558e+58 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 1.791697211e+71
AT LEVEL 0 5   UP: Norm before smooth 7.199454794e+71
AT LEVEL 0 5   UP: Norm after  smooth 1.185346574e+71
AT LEVEL 0 4   UP: Norm before smooth 6.00123975e+72
AT LEVEL 0 4   UP: Norm after  smooth 1.881834298e+71
AT LEVEL 0 3   UP: Norm before smooth 8.043939857e+73
AT LEVEL 0 3   UP: Norm after  smooth 1.004935323e+72
AT LEVEL 0 2   UP: Norm before smooth 2.631656375e+75
AT LEVEL 0 2   UP: Norm after  smooth 2.586624813e+73
AT LEVEL 0 1   UP: Norm before smooth 3.904610829e+77
AT LEVEL 0 1   UP: Norm after  smooth 2.456953261e+76
AT LEVEL 0 0   UP: Norm before smooth 5.355273183e+79
AT LEVEL 0 0   UP: Norm after  smooth 1.075401037e+87
MLMG: Iteration   6 Fine resid/bnorm = 1.533088493e+81
MLMG: Subtracting 1.325303872e+66 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 1.075401037e+87
AT LEVEL 0 0   DN: Norm after  smooth 1.429938741e+94
AT LEVEL 0 1   DN: Norm before smooth 4.579733586e+93
AT LEVEL 0 1   DN: Norm after  smooth 1.064483179e+92
AT LEVEL 0 2   DN: Norm before smooth 4.297496864e+91
AT LEVEL 0 2   DN: Norm after  smooth 9.923661028e+89
AT LEVEL 0 3   DN: Norm before smooth 4.50803986e+89
AT LEVEL 0 3   DN: Norm after  smooth 4.153081775e+88
AT LEVEL 0 4   DN: Norm before smooth 2.274431053e+88
AT LEVEL 0 4   DN: Norm after  smooth 2.013966313e+87
AT LEVEL 0 5   DN: Norm before smooth 1.113152873e+87
AT LEVEL 0 5   DN: Norm after  smooth 1.244293532e+86
AT LEVEL 0 6   DN: Norm before bottom 8.437610373e+85
MLMG: Subtracting -3.705752509e+72 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 2.101825349e+85
AT LEVEL 0 5   UP: Norm before smooth 8.445621554e+85
AT LEVEL 0 5   UP: Norm after  smooth 1.390520374e+85
AT LEVEL 0 4   UP: Norm before smooth 7.040005282e+86
AT LEVEL 0 4   UP: Norm after  smooth 2.207564429e+85
AT LEVEL 0 3   UP: Norm before smooth 9.436280076e+87
AT LEVEL 0 3   UP: Norm after  smooth 1.178881411e+86
AT LEVEL 0 2   UP: Norm before smooth 3.087174576e+89
AT LEVEL 0 2   UP: Norm after  smooth 3.034348418e+87
AT LEVEL 0 1   UP: Norm before smooth 4.580467039e+91
AT LEVEL 0 1   UP: Norm after  smooth 2.882231782e+90
AT LEVEL 0 0   UP: Norm before smooth 6.282227184e+93
AT LEVEL 0 0   UP: Norm after  smooth 1.261544163e+101
MLMG: Iteration   7 Fine resid/bnorm = 1.798453576e+95
MLMG: Subtracting 7.310537743e+79 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 1.261544163e+101
AT LEVEL 0 0   DN: Norm after  smooth 1.677449446e+108
AT LEVEL 0 1   DN: Norm before smooth 5.372448024e+107
AT LEVEL 0 1   DN: Norm after  smooth 1.248736514e+106
AT LEVEL 0 2   DN: Norm before smooth 5.041358434e+105
AT LEVEL 0 2   DN: Norm after  smooth 1.164136561e+104
AT LEVEL 0 3   DN: Norm before smooth 5.2883447e+103
AT LEVEL 0 3   DN: Norm after  smooth 4.871946274e+102
AT LEVEL 0 4   DN: Norm before smooth 2.668116472e+102
AT LEVEL 0 4   DN: Norm after  smooth 2.362567415e+101
AT LEVEL 0 5   DN: Norm before smooth 1.305830534e+101
AT LEVEL 0 5   DN: Norm after  smooth 1.459670569e+100
AT LEVEL 0 6   DN: Norm before bottom 9.89809174e+99
MLMG: Subtracting -8.632488262e+87 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 2.465634132e+99
AT LEVEL 0 5   UP: Norm before smooth 9.907489591e+99
AT LEVEL 0 5   UP: Norm after  smooth 1.631208081e+99
AT LEVEL 0 4   UP: Norm before smooth 8.258572635e+100
AT LEVEL 0 4   UP: Norm after  smooth 2.589675781e+99
AT LEVEL 0 3   UP: Norm before smooth 1.1069623e+102
AT LEVEL 0 3   UP: Norm after  smooth 1.382936144e+100
AT LEVEL 0 2   UP: Norm before smooth 3.621539254e+103
AT LEVEL 0 2   UP: Norm after  smooth 3.55956932e+101
AT LEVEL 0 1   UP: Norm before smooth 5.373308431e+105
AT LEVEL 0 1   UP: Norm after  smooth 3.381122537e+104
AT LEVEL 0 0   UP: Norm before smooth 7.369629344e+107
AT LEVEL 0 0   UP: Norm after  smooth 1.479907143e+115
MLMG: Iteration   8 Fine resid/bnorm = 2.109751186e+109
MLMG: Subtracting -1.066753182e+94 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 1.479907143e+115
AT LEVEL 0 0   DN: Norm after  smooth 1.967802229e+122
AT LEVEL 0 1   DN: Norm before smooth 6.30237485e+121
AT LEVEL 0 1   DN: Norm after  smooth 1.464882594e+120
AT LEVEL 0 2   DN: Norm before smooth 5.913976359e+119
AT LEVEL 0 2   DN: Norm after  smooth 1.365639082e+118
AT LEVEL 0 3   DN: Norm before smooth 6.203713928e+117
AT LEVEL 0 3   DN: Norm after  smooth 5.715240339e+116
AT LEVEL 0 4   DN: Norm before smooth 3.129945618e+116
AT LEVEL 0 4   DN: Norm after  smooth 2.771508517e+115
AT LEVEL 0 5   DN: Norm before smooth 1.531859122e+115
AT LEVEL 0 5   DN: Norm after  smooth 1.712327609e+114
AT LEVEL 0 6   DN: Norm before bottom 1.161137049e+114
MLMG: Subtracting 6.610927936e+101 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 2.89241524e+113
AT LEVEL 0 5   UP: Norm before smooth 1.162239503e+114
AT LEVEL 0 5   UP: Norm after  smooth 1.913556863e+113
AT LEVEL 0 4   UP: Norm before smooth 9.688064033e+114
AT LEVEL 0 4   UP: Norm after  smooth 3.037927485e+113
AT LEVEL 0 3   UP: Norm before smooth 1.298568423e+116
AT LEVEL 0 3   UP: Norm after  smooth 1.622311084e+114
AT LEVEL 0 2   UP: Norm before smooth 4.248398087e+117
AT LEVEL 0 2   UP: Norm after  smooth 4.175701665e+115
AT LEVEL 0 1   UP: Norm before smooth 6.303384187e+119
AT LEVEL 0 1   UP: Norm after  smooth 3.966367203e+118
AT LEVEL 0 0   UP: Norm before smooth 8.645251926e+121
AT LEVEL 0 0   UP: Norm after  smooth 1.736066969e+129
MLMG: Iteration   9 Fine resid/bnorm = 2.474931866e+123
MLMG: Subtracting -1.717491975e+108 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 1.736066969e+129
AT LEVEL 0 0   DN: Norm after  smooth 2.308412706e+136
AT LEVEL 0 1   DN: Norm before smooth 7.393264406e+135
AT LEVEL 0 1   DN: Norm after  smooth 1.718441794e+134
AT LEVEL 0 2   DN: Norm before smooth 6.937637311e+133
AT LEVEL 0 2   DN: Norm after  smooth 1.602020041e+132
AT LEVEL 0 3   DN: Norm before smooth 7.277526086e+131
AT LEVEL 0 3   DN: Norm after  smooth 6.704501712e+130
AT LEVEL 0 4   DN: Norm before smooth 3.671713612e+130
AT LEVEL 0 4   DN: Norm after  smooth 3.251233981e+129
AT LEVEL 0 5   DN: Norm before smooth 1.797011411e+129
AT LEVEL 0 5   DN: Norm after  smooth 2.008717518e+128
AT LEVEL 0 6   DN: Norm before bottom 1.362120378e+128
MLMG: Subtracting 2.168723132e+115 on level (0, 6)
MLMG: Bottom solve failed.
AT LEVEL 0 6   UP: Norm after  bottom 3.393068669e+127
AT LEVEL 0 5   UP: Norm before smooth 1.363413658e+128
AT LEVEL 0 5   UP: Norm after  smooth 2.24477791e+127
AT LEVEL 0 4   UP: Norm before smooth 1.13649887e+129
AT LEVEL 0 4   UP: Norm after  smooth 3.563767894e+127
AT LEVEL 0 3   UP: Norm before smooth 1.52334e+130
AT LEVEL 0 3   UP: Norm after  smooth 1.903119868e+128
AT LEVEL 0 2   UP: Norm before smooth 4.98376106e+131
AT LEVEL 0 2   UP: Norm after  smooth 4.898481481e+129
AT LEVEL 0 1   UP: Norm before smooth 7.394448451e+133
AT LEVEL 0 1   UP: Norm after  smooth 4.652912936e+132
AT LEVEL 0 0   UP: Norm before smooth 1.014167435e+136
AT LEVEL 0 0   UP: Norm after  smooth 2.036565967e+143
MLMG: Iteration  10 Fine resid/bnorm = 2.903322336e+137
MLMG: Subtracting -8.686132732e+121 on level (0, 0)
AT LEVEL 0 0   DN: Norm before smooth 2.036565967e+143
AT LEVEL 0 0   DN: Norm after  smooth 2.70798007e+150
AT LEVEL 0 1   DN: Norm before smooth 8.672978025e+149
AT LEVEL 0 1   DN: Norm after  smooth 2.015890018e+148
AT LEVEL 0 2   DN: Norm before smooth 8.138485605e+147
AT LEVEL 0 2   DN: Norm after  smooth 1.879316611e+146
AT LEVEL 0 3   DN: Norm before smooth 8.537206347e+145
AT LEVEL 0 3   DN: Norm after  smooth 7.864996139e+144
AT LEVEL 0 4   DN: Norm before smooth 4.307257218e+144
AT LEVEL 0 4   DN: Norm after  smooth 3.813995999e+143
AT LEVEL 0 5   DN: Norm before smooth 2.108059394e+143
AT LEVEL 0 5   DN: Norm after  smooth 2.356410096e+142
AT LEVEL 0 6   DN: Norm before bottom 1.597892279e+142
MLMG: Subtracting -2.10021712e+129 on level (0, 6)
AT LEVEL 0 6   UP: Norm after  bottom 0
AT LEVEL 0 5   UP: Norm before smooth 0
AT LEVEL 0 5   UP: Norm after  smooth 0
AT LEVEL 0 4   UP: Norm before smooth 0
AT LEVEL 0 4   UP: Norm after  smooth 0
AT LEVEL 0 3   UP: Norm before smooth 0
AT LEVEL 0 3   UP: Norm after  smooth 0
AT LEVEL 0 2   UP: Norm before smooth 0
AT LEVEL 0 2   UP: Norm after  smooth 0
AT LEVEL 0 1   UP: Norm before smooth 0
AT LEVEL 0 1   UP: Norm after  smooth 0
AT LEVEL 0 0   UP: Norm before smooth 0
AT LEVEL 0 0   UP: Norm after  smooth 0
MLMG: Iteration  11 Fine resid/bnorm = 0
MLMG: Final Iter. 11 resid, resid/bnorm = 0, 0
MLMG: Timers: Solve = 2.045967801 Iter = 2.037279753 Bottom = 0.003868574
 >> After projection:
  * On lev 0 max(abs(rhs)) = 0

Projection::level_project(): lev: 0, time: 2.065374957
NavierStokes::advance(): after velocity update
max(abs(u/v))  = 0  0
max(abs(gpx/gpy/p)) = 0  0  0
[Level 0 step 6] Advanced 131072 cells

[STEP 6] Coarse TimeStep time: 2.447419548
[STEP 6] FAB kilobyte spread across MPI nodes: [226900 ... 226900]

STEP = 6 TIME = 0.0004388865319 DT = 9.161053351e-05

PLOTFILE: file = plt00006
Write plotfile time = 0.066868738  seconds

NavierStokesBase::estTimeStep() failed to provide a good timestep (probably because initial velocity field is zero with no external forcing).
Use ns.init_dt to provide a reasonable timestep on coarsest level.
Note that ns.init_shrink will be applied to init_dt.
amrex::Abort::0::
 !!!
SIGABRT
See Backtrace.0 file for details

I don't know if there's anything obvious that stands out here, the initial iteration of the MG solver doesn't look unusual compared to the other time steps, but starts blowing up after iteration 1 or 2 whereas it converges in the previous time step.

asalmgren commented 4 years ago

It would be helpful if I could play with this -- can you send me whatever info I need to try to run this locally?

On Wed, Jul 1, 2020 at 10:59 AM dcoveney notifications@github.com wrote:

Thanks for this.

I just tried this with proj.proj_tol = 1.0e-10 and I had a similar crash. Here is the terminal output for the last two time steps:

[STEP 4] Coarse TimeStep time: 1.709251631 [STEP 4] FAB kilobyte spread across MPI nodes: [226900 ... 226900]

STEP = 4 TIME = 0.0002639936952 DT = 7.571118472e-05

PLOTFILE: file = plt00004 Write plotfile time = 0.057479739 seconds

estTimeStep :: LEV = 0 UMAX = 1.918219757 1.880784946 estimated timestep: dt = 0.0002036393373 [Level 0 step 5] ADVANCE with dt = 8.328230319e-05 Advancing grids at level 0 : starting time = 0.0002639936952 with dt = 8.328230319e-05 NavierStokes::advance(): before velocity update: max(abs(u/v)) = 1.918219757 1.880784946 max(abs(gpx/gpy/p)) = 56047.27994 12870.78883 4662.571817 ... predict edge velocities ... mac_projection ... mac_project at level 0 SUM of DIV(U_edge) = -6656 NavierStokesBase:mac_project(): lev: 0, time: 0.193923521 ... advect velocities ... advect scalars ... update scalars ... update scalars ... update velocities NavierStokes:velocity_diffusion_update(): lev: 0, time: 1.830003384e-07 NavierStokes::advance(): before nodal projection max(abs(u/v)) = 1.92277004 1.834439165 max(abs(gpx/gpy/p)) = 56047.27994 12870.78883 4671.796138 ... Projection::level_project() at level 0 Nodal Projection:

Before projection:

  • On lev 0 max(abs(rhs)) = 726976.8117

MLMG: Subtracting -609.7483866 from rhs MLMG: # of AMR levels: 1

of MG levels on the coarsest AMR level: 7

MLMG: Initial rhs = 727586.5601 MLMG: Initial residual (resid0) = 727586.5601 MLMG: Subtracting 4.303918333e-13 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 727586.5601 AT LEVEL 0 0 DN: Norm after smooth 5781.929765 AT LEVEL 0 1 DN: Norm before smooth 3885.712465 AT LEVEL 0 1 DN: Norm after smooth 1551.668289 AT LEVEL 0 2 DN: Norm before smooth 1053.841467 AT LEVEL 0 2 DN: Norm after smooth 667.2690317 AT LEVEL 0 3 DN: Norm before smooth 435.7146835 AT LEVEL 0 3 DN: Norm after smooth 125.2013589 AT LEVEL 0 4 DN: Norm before smooth 82.91739746 AT LEVEL 0 4 DN: Norm after smooth 51.5325694 AT LEVEL 0 5 DN: Norm before smooth 34.58015972 AT LEVEL 0 5 DN: Norm after smooth 4.304889328 AT LEVEL 0 6 DN: Norm before bottom 2.774173854 MLMG: Subtracting 2.368244489e-14 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 0.04542755851 AT LEVEL 0 5 UP: Norm before smooth 6.071179849 AT LEVEL 0 5 UP: Norm after smooth 0.04217304486 AT LEVEL 0 4 UP: Norm before smooth 46.36686106 AT LEVEL 0 4 UP: Norm after smooth 1.943684422 AT LEVEL 0 3 UP: Norm before smooth 139.0154107 AT LEVEL 0 3 UP: Norm after smooth 7.230719772 AT LEVEL 0 2 UP: Norm before smooth 797.1871627 AT LEVEL 0 2 UP: Norm after smooth 22.78696314 AT LEVEL 0 1 UP: Norm before smooth 4421.163206 AT LEVEL 0 1 UP: Norm after smooth 68.45611824 AT LEVEL 0 0 UP: Norm before smooth 15103.67505 AT LEVEL 0 0 UP: Norm after smooth 308.9836274 MLMG: Iteration 1 Fine resid/bnorm = 0.0004246692344 MLMG: Subtracting 1.390484949e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 308.9836274 AT LEVEL 0 0 DN: Norm after smooth 514.7721261 AT LEVEL 0 1 DN: Norm before smooth 199.2470902 AT LEVEL 0 1 DN: Norm after smooth 50.36863492 AT LEVEL 0 2 DN: Norm before smooth 34.71612638 AT LEVEL 0 2 DN: Norm after smooth 26.71743829 AT LEVEL 0 3 DN: Norm before smooth 17.43863803 AT LEVEL 0 3 DN: Norm after smooth 8.102352416 AT LEVEL 0 4 DN: Norm before smooth 5.507201959 AT LEVEL 0 4 DN: Norm after smooth 2.08050573 AT LEVEL 0 5 DN: Norm before smooth 1.330782877 AT LEVEL 0 5 DN: Norm after smooth 0.1863928778 AT LEVEL 0 6 DN: Norm before bottom 0.1244828817 MLMG: Subtracting -3.638582491e-16 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 0.0007790313336 AT LEVEL 0 5 UP: Norm before smooth 0.1648968546 AT LEVEL 0 5 UP: Norm after smooth 0.0005600761544 AT LEVEL 0 4 UP: Norm before smooth 1.733872978 AT LEVEL 0 4 UP: Norm after smooth 0.07434496825 AT LEVEL 0 3 UP: Norm before smooth 7.641405832 AT LEVEL 0 3 UP: Norm after smooth 0.3429656375 AT LEVEL 0 2 UP: Norm before smooth 30.35014544 AT LEVEL 0 2 UP: Norm after smooth 1.148433919 AT LEVEL 0 1 UP: Norm before smooth 174.7750096 AT LEVEL 0 1 UP: Norm after smooth 2.047428595 AT LEVEL 0 0 UP: Norm before smooth 705.4816146 AT LEVEL 0 0 UP: Norm after smooth 89.52163722 MLMG: Iteration 2 Fine resid/bnorm = 0.0001230391573 MLMG: Subtracting 1.23019997e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 89.52163722 AT LEVEL 0 0 DN: Norm after smooth 96.25353282 AT LEVEL 0 1 DN: Norm before smooth 37.01451762 AT LEVEL 0 1 DN: Norm after smooth 2.861900667 AT LEVEL 0 2 DN: Norm before smooth 1.574315764 AT LEVEL 0 2 DN: Norm after smooth 1.168929959 AT LEVEL 0 3 DN: Norm before smooth 0.7623740571 AT LEVEL 0 3 DN: Norm after smooth 0.2727603352 AT LEVEL 0 4 DN: Norm before smooth 0.1816480787 AT LEVEL 0 4 DN: Norm after smooth 0.1009015168 AT LEVEL 0 5 DN: Norm before smooth 0.06322336384 AT LEVEL 0 5 DN: Norm after smooth 0.009795415901 AT LEVEL 0 6 DN: Norm before bottom 0.006116800093 MLMG: Subtracting -7.155734338e-18 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 0.0002367386316 AT LEVEL 0 5 UP: Norm before smooth 0.008324693203 AT LEVEL 0 5 UP: Norm after smooth 0.0001496849684 AT LEVEL 0 4 UP: Norm before smooth 0.08471655633 AT LEVEL 0 4 UP: Norm after smooth 0.003627525886 AT LEVEL 0 3 UP: Norm before smooth 0.3366393005 AT LEVEL 0 3 UP: Norm after smooth 0.01577659517 AT LEVEL 0 2 UP: Norm before smooth 1.334980616 AT LEVEL 0 2 UP: Norm after smooth 0.0569644283 AT LEVEL 0 1 UP: Norm before smooth 7.458849476 AT LEVEL 0 1 UP: Norm after smooth 0.110163214 AT LEVEL 0 0 UP: Norm before smooth 44.52699824 AT LEVEL 0 0 UP: Norm after smooth 13.93153677 MLMG: Iteration 3 Fine resid/bnorm = 1.914760049e-05 MLMG: Subtracting 1.212006027e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 13.93153678 AT LEVEL 0 0 DN: Norm after smooth 13.7102312 AT LEVEL 0 1 DN: Norm before smooth 5.263212122 AT LEVEL 0 1 DN: Norm after smooth 0.2208144174 AT LEVEL 0 2 DN: Norm before smooth 0.1069266951 AT LEVEL 0 2 DN: Norm after smooth 0.05662250156 AT LEVEL 0 3 DN: Norm before smooth 0.03523740058 AT LEVEL 0 3 DN: Norm after smooth 0.01843526223 AT LEVEL 0 4 DN: Norm before smooth 0.01258629828 AT LEVEL 0 4 DN: Norm after smooth 0.005108690705 AT LEVEL 0 5 DN: Norm before smooth 0.003352680777 AT LEVEL 0 5 DN: Norm after smooth 0.0004701936047 AT LEVEL 0 6 DN: Norm before bottom 0.0003149877577 MLMG: Subtracting -3.904821887e-19 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 1.027752768e-05 AT LEVEL 0 5 UP: Norm before smooth 0.0004120262623 AT LEVEL 0 5 UP: Norm after smooth 6.043355783e-06 AT LEVEL 0 4 UP: Norm before smooth 0.004386870411 AT LEVEL 0 4 UP: Norm after smooth 0.0001879987059 AT LEVEL 0 3 UP: Norm before smooth 0.0187712218 AT LEVEL 0 3 UP: Norm after smooth 0.0008508209144 AT LEVEL 0 2 UP: Norm before smooth 0.07193143351 AT LEVEL 0 2 UP: Norm after smooth 0.003124074591 AT LEVEL 0 1 UP: Norm before smooth 0.3392636185 AT LEVEL 0 1 UP: Norm after smooth 0.006303778595 AT LEVEL 0 0 UP: Norm before smooth 6.215739699 AT LEVEL 0 0 UP: Norm after smooth 1.887033965 MLMG: Iteration 4 Fine resid/bnorm = 2.593552534e-06 MLMG: Subtracting 1.376882928e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 1.887033967 AT LEVEL 0 0 DN: Norm after smooth 1.8073092 AT LEVEL 0 1 DN: Norm before smooth 0.6934140883 AT LEVEL 0 1 DN: Norm after smooth 0.02103211361 AT LEVEL 0 2 DN: Norm before smooth 0.01002116655 AT LEVEL 0 2 DN: Norm after smooth 0.003557997337 AT LEVEL 0 3 DN: Norm before smooth 0.002077650965 AT LEVEL 0 3 DN: Norm after smooth 0.000736779751 AT LEVEL 0 4 DN: Norm before smooth 0.0004711199177 AT LEVEL 0 4 DN: Norm after smooth 0.0002628469375 AT LEVEL 0 5 DN: Norm before smooth 0.0001686099859 AT LEVEL 0 5 DN: Norm after smooth 2.814042791e-05 AT LEVEL 0 6 DN: Norm before bottom 1.760594174e-05 MLMG: Subtracting 1.196434038e-20 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 1.245030424e-06 AT LEVEL 0 5 UP: Norm before smooth 2.131731721e-05 AT LEVEL 0 5 UP: Norm after smooth 8.670749792e-07 AT LEVEL 0 4 UP: Norm before smooth 0.0002241844423 AT LEVEL 0 4 UP: Norm after smooth 9.600649717e-06 AT LEVEL 0 3 UP: Norm before smooth 0.0009122373209 AT LEVEL 0 3 UP: Norm after smooth 4.21707361e-05 AT LEVEL 0 2 UP: Norm before smooth 0.003935915679 AT LEVEL 0 2 UP: Norm after smooth 0.0001599380214 AT LEVEL 0 1 UP: Norm before smooth 0.01504381927 AT LEVEL 0 1 UP: Norm after smooth 0.0003978510211 AT LEVEL 0 0 UP: Norm before smooth 0.814837668 AT LEVEL 0 0 UP: Norm after smooth 0.2438598809 MLMG: Iteration 5 Fine resid/bnorm = 3.351627087e-07 MLMG: Subtracting 1.362447237e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 0.2438598823 AT LEVEL 0 0 DN: Norm after smooth 0.230853851 AT LEVEL 0 1 DN: Norm before smooth 0.08855015765 AT LEVEL 0 1 DN: Norm after smooth 0.002231679449 AT LEVEL 0 2 DN: Norm before smooth 0.001050100473 AT LEVEL 0 2 DN: Norm after smooth 0.0002306821717 AT LEVEL 0 3 DN: Norm before smooth 0.0001327772516 AT LEVEL 0 3 DN: Norm after smooth 4.575058407e-05 AT LEVEL 0 4 DN: Norm before smooth 3.13573135e-05 AT LEVEL 0 4 DN: Norm after smooth 1.452837284e-05 AT LEVEL 0 5 DN: Norm before smooth 9.087575767e-06 AT LEVEL 0 5 DN: Norm after smooth 1.457197901e-06 AT LEVEL 0 6 DN: Norm before bottom 9.800742062e-07 MLMG: Subtracting -3.67268192e-21 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 8.063609705e-08 AT LEVEL 0 5 UP: Norm before smooth 1.089846713e-06 AT LEVEL 0 5 UP: Norm after smooth 5.223839905e-08 AT LEVEL 0 4 UP: Norm before smooth 1.187499724e-05 AT LEVEL 0 4 UP: Norm after smooth 5.087963902e-07 AT LEVEL 0 3 UP: Norm before smooth 5.100428209e-05 AT LEVEL 0 3 UP: Norm after smooth 2.251302182e-06 AT LEVEL 0 2 UP: Norm before smooth 0.0002161870219 AT LEVEL 0 2 UP: Norm after smooth 8.56632633e-06 AT LEVEL 0 1 UP: Norm before smooth 0.0007693354972 AT LEVEL 0 1 UP: Norm after smooth 2.648630204e-05 AT LEVEL 0 0 UP: Norm before smooth 0.1037347356 AT LEVEL 0 0 UP: Norm after smooth 0.03090814427 MLMG: Iteration 6 Fine resid/bnorm = 4.248037238e-08 MLMG: Subtracting 1.200004533e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 0.03090814801 AT LEVEL 0 0 DN: Norm after smooth 0.02912779376 AT LEVEL 0 1 DN: Norm before smooth 0.01117165159 AT LEVEL 0 1 DN: Norm after smooth 0.0002590554468 AT LEVEL 0 2 DN: Norm before smooth 0.0001210958103 AT LEVEL 0 2 DN: Norm after smooth 1.839192302e-05 AT LEVEL 0 3 DN: Norm before smooth 1.037546348e-05 AT LEVEL 0 3 DN: Norm after smooth 2.083599409e-06 AT LEVEL 0 4 DN: Norm before smooth 1.334406961e-06 AT LEVEL 0 4 DN: Norm after smooth 7.115565523e-07 AT LEVEL 0 5 DN: Norm before smooth 4.699012154e-07 AT LEVEL 0 5 DN: Norm after smooth 9.256460306e-08 AT LEVEL 0 6 DN: Norm before bottom 5.808612107e-08 MLMG: Subtracting 4.063524759e-22 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 7.236287323e-09 AT LEVEL 0 5 UP: Norm before smooth 5.749241134e-08 AT LEVEL 0 5 UP: Norm after smooth 5.078890277e-09 AT LEVEL 0 4 UP: Norm before smooth 6.215994981e-07 AT LEVEL 0 4 UP: Norm after smooth 2.662380058e-08 AT LEVEL 0 3 UP: Norm before smooth 2.569960313e-06 AT LEVEL 0 3 UP: Norm after smooth 1.17776129e-07 AT LEVEL 0 2 UP: Norm before smooth 1.18413588e-05 AT LEVEL 0 2 UP: Norm after smooth 4.347492031e-07 AT LEVEL 0 1 UP: Norm before smooth 8.915226982e-05 AT LEVEL 0 1 UP: Norm after smooth 1.760088333e-06 AT LEVEL 0 0 UP: Norm before smooth 0.01307474151 AT LEVEL 0 0 UP: Norm after smooth 0.003886744124 MLMG: Iteration 7 Fine resid/bnorm = 5.341969347e-09 MLMG: Subtracting 1.21705513e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 0.003886745103 AT LEVEL 0 0 DN: Norm after smooth 0.003655464318 AT LEVEL 0 1 DN: Norm before smooth 0.001401952173 AT LEVEL 0 1 DN: Norm after smooth 3.123344065e-05 AT LEVEL 0 2 DN: Norm before smooth 1.45491972e-05 AT LEVEL 0 2 DN: Norm after smooth 1.698448809e-06 AT LEVEL 0 3 DN: Norm before smooth 9.418170222e-07 AT LEVEL 0 3 DN: Norm after smooth 1.418625994e-07 AT LEVEL 0 4 DN: Norm before smooth 8.70467465e-08 AT LEVEL 0 4 DN: Norm after smooth 4.485896078e-08 AT LEVEL 0 5 DN: Norm before smooth 2.738667329e-08 AT LEVEL 0 5 DN: Norm after smooth 5.385651795e-09 AT LEVEL 0 6 DN: Norm before bottom 3.642732964e-09 MLMG: Subtracting 4.34592939e-24 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 5.639027261e-10 AT LEVEL 0 5 UP: Norm before smooth 2.999543576e-09 AT LEVEL 0 5 UP: Norm after smooth 3.753608052e-10 AT LEVEL 0 4 UP: Norm before smooth 3.335717624e-08 AT LEVEL 0 4 UP: Norm after smooth 1.428924253e-09 AT LEVEL 0 3 UP: Norm before smooth 1.434681762e-07 AT LEVEL 0 3 UP: Norm after smooth 6.512329949e-09 AT LEVEL 0 2 UP: Norm before smooth 6.508552765e-07 AT LEVEL 0 2 UP: Norm after smooth 2.255522747e-08 AT LEVEL 0 1 UP: Norm before smooth 1.070303661e-05 AT LEVEL 0 1 UP: Norm after smooth 1.190010584e-07 AT LEVEL 0 0 UP: Norm before smooth 0.001639860996 AT LEVEL 0 0 UP: Norm after smooth 0.0004870777249 MLMG: Iteration 8 Fine resid/bnorm = 6.694449144e-10 MLMG: Subtracting 1.296554173e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 0.0004870791211 AT LEVEL 0 0 DN: Norm after smooth 0.0004577057848 AT LEVEL 0 1 DN: Norm before smooth 0.0001755371309 AT LEVEL 0 1 DN: Norm after smooth 3.843425417e-06 AT LEVEL 0 2 DN: Norm before smooth 1.787593864e-06 AT LEVEL 0 2 DN: Norm after smooth 1.809471265e-07 AT LEVEL 0 3 DN: Norm before smooth 9.904325601e-08 AT LEVEL 0 3 DN: Norm after smooth 1.153612554e-08 AT LEVEL 0 4 DN: Norm before smooth 6.645044494e-09 AT LEVEL 0 4 DN: Norm after smooth 2.17561113e-09 AT LEVEL 0 5 DN: Norm before smooth 1.386879828e-09 AT LEVEL 0 5 DN: Norm after smooth 3.550225092e-10 AT LEVEL 0 6 DN: Norm before bottom 2.235719467e-10 MLMG: Subtracting 5.10687093e-24 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 4.28276965e-11 AT LEVEL 0 5 UP: Norm before smooth 1.601541119e-10 AT LEVEL 0 5 UP: Norm after smooth 2.962254078e-11 AT LEVEL 0 4 UP: Norm before smooth 1.773505625e-09 AT LEVEL 0 4 UP: Norm after smooth 7.597946366e-11 AT LEVEL 0 3 UP: Norm before smooth 7.416892851e-09 AT LEVEL 0 3 UP: Norm after smooth 3.299500371e-10 AT LEVEL 0 2 UP: Norm before smooth 6.044005136e-08 AT LEVEL 0 2 UP: Norm after smooth 1.466266586e-09 AT LEVEL 0 1 UP: Norm before smooth 1.316416734e-06 AT LEVEL 0 1 UP: Norm after smooth 8.236023845e-09 AT LEVEL 0 0 UP: Norm before smooth 0.0002052859254 AT LEVEL 0 0 UP: Norm after smooth 6.095071136e-05 MLMG: Iteration 9 Fine resid/bnorm = 8.377293097e-11 MLMG: Final Iter. 9 resid, resid/bnorm = 6.095205867e-05, 8.377293097e-11 MLMG: Timers: Solve = 1.756000754 Iter = 1.746957118 Bottom = 0.003314982

After projection:

  • On lev 0 max(abs(rhs)) = 167063.8425

Projection::level_project(): lev: 0, time: 1.778533854 NavierStokes::advance(): after velocity update max(abs(u/v)) = 1.92228911 1.800679662 max(abs(gpx/gpy/p)) = 49053.33771 13697.83594 4658.531877 [Level 0 step 5] Advanced 131072 cells

[STEP 5] Coarse TimeStep time: 2.167339015 [STEP 5] FAB kilobyte spread across MPI nodes: [226900 ... 226900]

STEP = 5 TIME = 0.0003472759984 DT = 8.328230319e-05

PLOTFILE: file = plt00005 Write plotfile time = 0.056482354 seconds

estTimeStep :: LEV = 0 UMAX = 1.92228911 1.800679662 estimated timestep: dt = 0.0002032082468 [Level 0 step 6] ADVANCE with dt = 9.161053351e-05 Advancing grids at level 0 : starting time = 0.0003472759984 with dt = 9.161053351e-05 NavierStokes::advance(): before velocity update: max(abs(u/v)) = 1.92228911 1.800679662 max(abs(gpx/gpy/p)) = 49053.33771 13697.83594 4658.531877 ... predict edge velocities ... mac_projection ... mac_project at level 0 SUM of DIV(U_edge) = -6656.00000000001 NavierStokesBase:mac_project(): lev: 0, time: 0.193445177 ... advect velocities ... advect scalars ... update scalars ... update scalars ... update velocities NavierStokes:velocity_diffusion_update(): lev: 0, time: 2.229999154e-07 NavierStokes::advance(): before nodal projection max(abs(u/v)) = 1.927049302 1.745457294 max(abs(gpx/gpy/p)) = 49053.33771 13697.83594 4662.571817 ... Projection::level_project() at level 0 Nodal Projection:

Before projection:

  • On lev 0 max(abs(rhs)) = 700906.1935

MLMG: Subtracting -554.3167151 from rhs MLMG: # of AMR levels: 1

of MG levels on the coarsest AMR level: 7

MLMG: Initial rhs = 701460.5102 MLMG: Initial residual (resid0) = 701460.5102 MLMG: Subtracting -1.641131675e-12 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 701460.5102 AT LEVEL 0 0 DN: Norm after smooth 5486875205 AT LEVEL 0 1 DN: Norm before smooth 1757307913 AT LEVEL 0 1 DN: Norm after smooth 40845685.58 AT LEVEL 0 2 DN: Norm before smooth 16490086.69 AT LEVEL 0 2 DN: Norm after smooth 380767.8523 AT LEVEL 0 3 DN: Norm before smooth 172970.1855 AT LEVEL 0 3 DN: Norm after smooth 15917.43893 AT LEVEL 0 4 DN: Norm before smooth 8716.572349 AT LEVEL 0 4 DN: Norm after smooth 773.881569 AT LEVEL 0 5 DN: Norm before smooth 433.3378402 AT LEVEL 0 5 DN: Norm after smooth 51.27691564 AT LEVEL 0 6 DN: Norm before bottom 34.7657818 MLMG: Subtracting 1.203807887e-11 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 8.042035863 AT LEVEL 0 5 UP: Norm before smooth 32.25063201 AT LEVEL 0 5 UP: Norm after smooth 5.32865551 AT LEVEL 0 4 UP: Norm before smooth 270.7536296 AT LEVEL 0 4 UP: Norm after smooth 7.061239781 AT LEVEL 0 3 UP: Norm before smooth 3549.853286 AT LEVEL 0 3 UP: Norm after smooth 37.22150095 AT LEVEL 0 2 UP: Norm before smooth 118371.245 AT LEVEL 0 2 UP: Norm after smooth 1137.091865 AT LEVEL 0 1 UP: Norm before smooth 17575399.23 AT LEVEL 0 1 UP: Norm after smooth 1106075.05 AT LEVEL 0 0 UP: Norm before smooth 2410577609 AT LEVEL 0 0 UP: Norm after smooth 4.840725006e+16 MLMG: Iteration 1 Fine resid/bnorm = 6.900923054e+10 MLMG: Subtracting 9.433302616e-05 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 4.840725006e+16 AT LEVEL 0 0 DN: Norm after smooth 6.436612934e+23 AT LEVEL 0 1 DN: Norm before smooth 2.061484984e+23 AT LEVEL 0 1 DN: Norm after smooth 4.791580227e+21 AT LEVEL 0 2 DN: Norm before smooth 1.934441183e+21 AT LEVEL 0 2 DN: Norm after smooth 4.466958135e+19 AT LEVEL 0 3 DN: Norm before smooth 2.029213339e+19 AT LEVEL 0 3 DN: Norm after smooth 1.869435319e+18 AT LEVEL 0 4 DN: Norm before smooth 1.023794371e+18 AT LEVEL 0 4 DN: Norm after smooth 9.065508364e+16 AT LEVEL 0 5 DN: Norm before smooth 5.010658131e+16 AT LEVEL 0 5 DN: Norm after smooth 5.600964301e+15 AT LEVEL 0 6 DN: Norm before bottom 3.798039069e+15 MLMG: Subtracting 143.8515625 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 9.460990068e+14 AT LEVEL 0 5 UP: Norm before smooth 3.801645159e+15 AT LEVEL 0 5 UP: Norm after smooth 6.25917822e+14 AT LEVEL 0 4 UP: Norm before smooth 3.168932189e+16 AT LEVEL 0 4 UP: Norm after smooth 9.936955581e+14 AT LEVEL 0 3 UP: Norm before smooth 4.247572335e+17 AT LEVEL 0 3 UP: Norm after smooth 5.306523363e+15 AT LEVEL 0 2 UP: Norm before smooth 1.389636299e+19 AT LEVEL 0 2 UP: Norm after smooth 1.365857551e+17 AT LEVEL 0 1 UP: Norm before smooth 2.061815135e+21 AT LEVEL 0 1 UP: Norm after smooth 1.297384974e+20 AT LEVEL 0 0 UP: Norm before smooth 2.827831961e+23 AT LEVEL 0 0 UP: Norm after smooth 5.678614925e+30 MLMG: Iteration 2 Fine resid/bnorm = 8.09541641e+24 MLMG: Subtracting -1.157167652e+10 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 5.678614925e+30 AT LEVEL 0 0 DN: Norm after smooth 7.550738004e+37 AT LEVEL 0 1 DN: Norm before smooth 2.41831118e+37 AT LEVEL 0 1 DN: Norm after smooth 5.620963587e+35 AT LEVEL 0 2 DN: Norm before smooth 2.269277135e+35 AT LEVEL 0 2 DN: Norm after smooth 5.240152065e+33 AT LEVEL 0 3 DN: Norm before smooth 2.380453576e+33 AT LEVEL 0 3 DN: Norm after smooth 2.193019288e+32 AT LEVEL 0 4 DN: Norm before smooth 1.201004805e+32 AT LEVEL 0 4 DN: Norm after smooth 1.063467374e+31 AT LEVEL 0 5 DN: Norm before smooth 5.877962084e+30 AT LEVEL 0 5 DN: Norm after smooth 6.570445426e+29 AT LEVEL 0 6 DN: Norm before bottom 4.455448578e+29 MLMG: Subtracting 2.814077691e+17 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 1.109861009e+29 AT LEVEL 0 5 UP: Norm before smooth 4.459678852e+29 AT LEVEL 0 5 UP: Norm after smooth 7.342590793e+28 AT LEVEL 0 4 UP: Norm before smooth 3.717448441e+30 AT LEVEL 0 4 UP: Norm after smooth 1.165696134e+29 AT LEVEL 0 3 UP: Norm before smooth 4.982792377e+31 AT LEVEL 0 3 UP: Norm after smooth 6.22503917e+29 AT LEVEL 0 2 UP: Norm before smooth 1.630170981e+33 AT LEVEL 0 2 UP: Norm after smooth 1.602276326e+31 AT LEVEL 0 1 UP: Norm before smooth 2.418698477e+35 AT LEVEL 0 1 UP: Norm after smooth 1.521951705e+34 AT LEVEL 0 0 UP: Norm before smooth 3.317306552e+37 AT LEVEL 0 0 UP: Norm after smooth 6.661536738e+44 MLMG: Iteration 3 Fine resid/bnorm = 9.496666799e+38 MLMG: Subtracting -1.682596856e+24 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 6.661536738e+44 AT LEVEL 0 0 DN: Norm after smooth 8.857709016e+51 AT LEVEL 0 1 DN: Norm before smooth 2.836901072e+51 AT LEVEL 0 1 DN: Norm after smooth 6.593906425e+49 AT LEVEL 0 2 DN: Norm before smooth 2.662070453e+49 AT LEVEL 0 2 DN: Norm after smooth 6.147179542e+47 AT LEVEL 0 3 DN: Norm before smooth 2.792490626e+47 AT LEVEL 0 3 DN: Norm after smooth 2.572612996e+46 AT LEVEL 0 4 DN: Norm before smooth 1.408888917e+46 AT LEVEL 0 4 DN: Norm after smooth 1.247544881e+45 AT LEVEL 0 5 DN: Norm before smooth 6.89538926e+44 AT LEVEL 0 5 DN: Norm after smooth 7.707735807e+43 AT LEVEL 0 6 DN: Norm before bottom 5.226650298e+43 MLMG: Subtracting 2.294850691e+31 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 1.301968875e+43 AT LEVEL 0 5 UP: Norm before smooth 5.231612798e+43 AT LEVEL 0 5 UP: Norm after smooth 8.613533224e+42 AT LEVEL 0 4 UP: Norm before smooth 4.360908371e+44 AT LEVEL 0 4 UP: Norm after smooth 1.367468604e+43 AT LEVEL 0 3 UP: Norm before smooth 5.845272996e+45 AT LEVEL 0 3 UP: Norm after smooth 7.302542552e+43 AT LEVEL 0 2 UP: Norm before smooth 1.912340249e+47 AT LEVEL 0 2 UP: Norm after smooth 1.879617257e+45 AT LEVEL 0 1 UP: Norm before smooth 2.837355407e+49 AT LEVEL 0 1 UP: Norm after smooth 1.785389101e+48 AT LEVEL 0 0 UP: Norm before smooth 3.891505193e+51 AT LEVEL 0 0 UP: Norm after smooth 7.81459428e+58 MLMG: Iteration 4 Fine resid/bnorm = 1.114046217e+53 MLMG: Subtracting 6.717107368e+37 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 7.81459428e+58 AT LEVEL 0 0 DN: Norm after smooth 1.039090602e+66 AT LEVEL 0 1 DN: Norm before smooth 3.327945452e+65 AT LEVEL 0 1 DN: Norm after smooth 7.735257713e+63 AT LEVEL 0 2 DN: Norm before smooth 3.122853082e+63 AT LEVEL 0 2 DN: Norm after smooth 7.211206059e+61 AT LEVEL 0 3 DN: Norm before smooth 3.275847921e+61 AT LEVEL 0 3 DN: Norm after smooth 3.017911271e+60 AT LEVEL 0 4 DN: Norm before smooth 1.652756069e+60 AT LEVEL 0 4 DN: Norm after smooth 1.4634847e+59 AT LEVEL 0 5 DN: Norm before smooth 8.088924762e+58 AT LEVEL 0 5 DN: Norm after smooth 9.041881854e+57 AT LEVEL 0 6 DN: Norm before bottom 6.131340729e+57 MLMG: Subtracting -2.434216717e+44 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 1.527329042e+57 AT LEVEL 0 5 UP: Norm before smooth 6.137162198e+57 AT LEVEL 0 5 UP: Norm after smooth 1.01044654e+57 AT LEVEL 0 4 UP: Norm before smooth 5.115745954e+58 AT LEVEL 0 4 UP: Norm after smooth 1.604166239e+57 AT LEVEL 0 3 UP: Norm before smooth 6.857041959e+59 AT LEVEL 0 3 UP: Norm after smooth 8.566552959e+57 AT LEVEL 0 2 UP: Norm before smooth 2.24335071e+61 AT LEVEL 0 2 UP: Norm after smooth 2.204963636e+59 AT LEVEL 0 1 UP: Norm before smooth 3.328478428e+63 AT LEVEL 0 1 UP: Norm after smooth 2.094425357e+62 AT LEVEL 0 0 UP: Norm before smooth 4.565092923e+65 AT LEVEL 0 0 UP: Norm after smooth 9.167236655e+72 MLMG: Iteration 5 Fine resid/bnorm = 1.306878509e+67 MLMG: Subtracting -8.094118466e+51 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 9.167236655e+72 AT LEVEL 0 0 DN: Norm after smooth 1.21894869e+80 AT LEVEL 0 1 DN: Norm before smooth 3.903985599e+79 AT LEVEL 0 1 DN: Norm after smooth 9.074167577e+77 AT LEVEL 0 2 DN: Norm before smooth 3.663393417e+77 AT LEVEL 0 2 DN: Norm after smooth 8.459406867e+75 AT LEVEL 0 3 DN: Norm before smooth 3.842870412e+75 AT LEVEL 0 3 DN: Norm after smooth 3.540287036e+74 AT LEVEL 0 4 DN: Norm before smooth 1.938834631e+74 AT LEVEL 0 4 DN: Norm after smooth 1.716801935e+73 AT LEVEL 0 5 DN: Norm before smooth 9.489051501e+72 AT LEVEL 0 5 DN: Norm after smooth 1.060695767e+72 AT LEVEL 0 6 DN: Norm before bottom 7.192625678e+71 MLMG: Subtracting -8.30658558e+58 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 1.791697211e+71 AT LEVEL 0 5 UP: Norm before smooth 7.199454794e+71 AT LEVEL 0 5 UP: Norm after smooth 1.185346574e+71 AT LEVEL 0 4 UP: Norm before smooth 6.00123975e+72 AT LEVEL 0 4 UP: Norm after smooth 1.881834298e+71 AT LEVEL 0 3 UP: Norm before smooth 8.043939857e+73 AT LEVEL 0 3 UP: Norm after smooth 1.004935323e+72 AT LEVEL 0 2 UP: Norm before smooth 2.631656375e+75 AT LEVEL 0 2 UP: Norm after smooth 2.586624813e+73 AT LEVEL 0 1 UP: Norm before smooth 3.904610829e+77 AT LEVEL 0 1 UP: Norm after smooth 2.456953261e+76 AT LEVEL 0 0 UP: Norm before smooth 5.355273183e+79 AT LEVEL 0 0 UP: Norm after smooth 1.075401037e+87 MLMG: Iteration 6 Fine resid/bnorm = 1.533088493e+81 MLMG: Subtracting 1.325303872e+66 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 1.075401037e+87 AT LEVEL 0 0 DN: Norm after smooth 1.429938741e+94 AT LEVEL 0 1 DN: Norm before smooth 4.579733586e+93 AT LEVEL 0 1 DN: Norm after smooth 1.064483179e+92 AT LEVEL 0 2 DN: Norm before smooth 4.297496864e+91 AT LEVEL 0 2 DN: Norm after smooth 9.923661028e+89 AT LEVEL 0 3 DN: Norm before smooth 4.50803986e+89 AT LEVEL 0 3 DN: Norm after smooth 4.153081775e+88 AT LEVEL 0 4 DN: Norm before smooth 2.274431053e+88 AT LEVEL 0 4 DN: Norm after smooth 2.013966313e+87 AT LEVEL 0 5 DN: Norm before smooth 1.113152873e+87 AT LEVEL 0 5 DN: Norm after smooth 1.244293532e+86 AT LEVEL 0 6 DN: Norm before bottom 8.437610373e+85 MLMG: Subtracting -3.705752509e+72 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 2.101825349e+85 AT LEVEL 0 5 UP: Norm before smooth 8.445621554e+85 AT LEVEL 0 5 UP: Norm after smooth 1.390520374e+85 AT LEVEL 0 4 UP: Norm before smooth 7.040005282e+86 AT LEVEL 0 4 UP: Norm after smooth 2.207564429e+85 AT LEVEL 0 3 UP: Norm before smooth 9.436280076e+87 AT LEVEL 0 3 UP: Norm after smooth 1.178881411e+86 AT LEVEL 0 2 UP: Norm before smooth 3.087174576e+89 AT LEVEL 0 2 UP: Norm after smooth 3.034348418e+87 AT LEVEL 0 1 UP: Norm before smooth 4.580467039e+91 AT LEVEL 0 1 UP: Norm after smooth 2.882231782e+90 AT LEVEL 0 0 UP: Norm before smooth 6.282227184e+93 AT LEVEL 0 0 UP: Norm after smooth 1.261544163e+101 MLMG: Iteration 7 Fine resid/bnorm = 1.798453576e+95 MLMG: Subtracting 7.310537743e+79 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 1.261544163e+101 AT LEVEL 0 0 DN: Norm after smooth 1.677449446e+108 AT LEVEL 0 1 DN: Norm before smooth 5.372448024e+107 AT LEVEL 0 1 DN: Norm after smooth 1.248736514e+106 AT LEVEL 0 2 DN: Norm before smooth 5.041358434e+105 AT LEVEL 0 2 DN: Norm after smooth 1.164136561e+104 AT LEVEL 0 3 DN: Norm before smooth 5.2883447e+103 AT LEVEL 0 3 DN: Norm after smooth 4.871946274e+102 AT LEVEL 0 4 DN: Norm before smooth 2.668116472e+102 AT LEVEL 0 4 DN: Norm after smooth 2.362567415e+101 AT LEVEL 0 5 DN: Norm before smooth 1.305830534e+101 AT LEVEL 0 5 DN: Norm after smooth 1.459670569e+100 AT LEVEL 0 6 DN: Norm before bottom 9.89809174e+99 MLMG: Subtracting -8.632488262e+87 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 2.465634132e+99 AT LEVEL 0 5 UP: Norm before smooth 9.907489591e+99 AT LEVEL 0 5 UP: Norm after smooth 1.631208081e+99 AT LEVEL 0 4 UP: Norm before smooth 8.258572635e+100 AT LEVEL 0 4 UP: Norm after smooth 2.589675781e+99 AT LEVEL 0 3 UP: Norm before smooth 1.1069623e+102 AT LEVEL 0 3 UP: Norm after smooth 1.382936144e+100 AT LEVEL 0 2 UP: Norm before smooth 3.621539254e+103 AT LEVEL 0 2 UP: Norm after smooth 3.55956932e+101 AT LEVEL 0 1 UP: Norm before smooth 5.373308431e+105 AT LEVEL 0 1 UP: Norm after smooth 3.381122537e+104 AT LEVEL 0 0 UP: Norm before smooth 7.369629344e+107 AT LEVEL 0 0 UP: Norm after smooth 1.479907143e+115 MLMG: Iteration 8 Fine resid/bnorm = 2.109751186e+109 MLMG: Subtracting -1.066753182e+94 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 1.479907143e+115 AT LEVEL 0 0 DN: Norm after smooth 1.967802229e+122 AT LEVEL 0 1 DN: Norm before smooth 6.30237485e+121 AT LEVEL 0 1 DN: Norm after smooth 1.464882594e+120 AT LEVEL 0 2 DN: Norm before smooth 5.913976359e+119 AT LEVEL 0 2 DN: Norm after smooth 1.365639082e+118 AT LEVEL 0 3 DN: Norm before smooth 6.203713928e+117 AT LEVEL 0 3 DN: Norm after smooth 5.715240339e+116 AT LEVEL 0 4 DN: Norm before smooth 3.129945618e+116 AT LEVEL 0 4 DN: Norm after smooth 2.771508517e+115 AT LEVEL 0 5 DN: Norm before smooth 1.531859122e+115 AT LEVEL 0 5 DN: Norm after smooth 1.712327609e+114 AT LEVEL 0 6 DN: Norm before bottom 1.161137049e+114 MLMG: Subtracting 6.610927936e+101 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 2.89241524e+113 AT LEVEL 0 5 UP: Norm before smooth 1.162239503e+114 AT LEVEL 0 5 UP: Norm after smooth 1.913556863e+113 AT LEVEL 0 4 UP: Norm before smooth 9.688064033e+114 AT LEVEL 0 4 UP: Norm after smooth 3.037927485e+113 AT LEVEL 0 3 UP: Norm before smooth 1.298568423e+116 AT LEVEL 0 3 UP: Norm after smooth 1.622311084e+114 AT LEVEL 0 2 UP: Norm before smooth 4.248398087e+117 AT LEVEL 0 2 UP: Norm after smooth 4.175701665e+115 AT LEVEL 0 1 UP: Norm before smooth 6.303384187e+119 AT LEVEL 0 1 UP: Norm after smooth 3.966367203e+118 AT LEVEL 0 0 UP: Norm before smooth 8.645251926e+121 AT LEVEL 0 0 UP: Norm after smooth 1.736066969e+129 MLMG: Iteration 9 Fine resid/bnorm = 2.474931866e+123 MLMG: Subtracting -1.717491975e+108 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 1.736066969e+129 AT LEVEL 0 0 DN: Norm after smooth 2.308412706e+136 AT LEVEL 0 1 DN: Norm before smooth 7.393264406e+135 AT LEVEL 0 1 DN: Norm after smooth 1.718441794e+134 AT LEVEL 0 2 DN: Norm before smooth 6.937637311e+133 AT LEVEL 0 2 DN: Norm after smooth 1.602020041e+132 AT LEVEL 0 3 DN: Norm before smooth 7.277526086e+131 AT LEVEL 0 3 DN: Norm after smooth 6.704501712e+130 AT LEVEL 0 4 DN: Norm before smooth 3.671713612e+130 AT LEVEL 0 4 DN: Norm after smooth 3.251233981e+129 AT LEVEL 0 5 DN: Norm before smooth 1.797011411e+129 AT LEVEL 0 5 DN: Norm after smooth 2.008717518e+128 AT LEVEL 0 6 DN: Norm before bottom 1.362120378e+128 MLMG: Subtracting 2.168723132e+115 on level (0, 6) MLMG: Bottom solve failed. AT LEVEL 0 6 UP: Norm after bottom 3.393068669e+127 AT LEVEL 0 5 UP: Norm before smooth 1.363413658e+128 AT LEVEL 0 5 UP: Norm after smooth 2.24477791e+127 AT LEVEL 0 4 UP: Norm before smooth 1.13649887e+129 AT LEVEL 0 4 UP: Norm after smooth 3.563767894e+127 AT LEVEL 0 3 UP: Norm before smooth 1.52334e+130 AT LEVEL 0 3 UP: Norm after smooth 1.903119868e+128 AT LEVEL 0 2 UP: Norm before smooth 4.98376106e+131 AT LEVEL 0 2 UP: Norm after smooth 4.898481481e+129 AT LEVEL 0 1 UP: Norm before smooth 7.394448451e+133 AT LEVEL 0 1 UP: Norm after smooth 4.652912936e+132 AT LEVEL 0 0 UP: Norm before smooth 1.014167435e+136 AT LEVEL 0 0 UP: Norm after smooth 2.036565967e+143 MLMG: Iteration 10 Fine resid/bnorm = 2.903322336e+137 MLMG: Subtracting -8.686132732e+121 on level (0, 0) AT LEVEL 0 0 DN: Norm before smooth 2.036565967e+143 AT LEVEL 0 0 DN: Norm after smooth 2.70798007e+150 AT LEVEL 0 1 DN: Norm before smooth 8.672978025e+149 AT LEVEL 0 1 DN: Norm after smooth 2.015890018e+148 AT LEVEL 0 2 DN: Norm before smooth 8.138485605e+147 AT LEVEL 0 2 DN: Norm after smooth 1.879316611e+146 AT LEVEL 0 3 DN: Norm before smooth 8.537206347e+145 AT LEVEL 0 3 DN: Norm after smooth 7.864996139e+144 AT LEVEL 0 4 DN: Norm before smooth 4.307257218e+144 AT LEVEL 0 4 DN: Norm after smooth 3.813995999e+143 AT LEVEL 0 5 DN: Norm before smooth 2.108059394e+143 AT LEVEL 0 5 DN: Norm after smooth 2.356410096e+142 AT LEVEL 0 6 DN: Norm before bottom 1.597892279e+142 MLMG: Subtracting -2.10021712e+129 on level (0, 6) AT LEVEL 0 6 UP: Norm after bottom 0 AT LEVEL 0 5 UP: Norm before smooth 0 AT LEVEL 0 5 UP: Norm after smooth 0 AT LEVEL 0 4 UP: Norm before smooth 0 AT LEVEL 0 4 UP: Norm after smooth 0 AT LEVEL 0 3 UP: Norm before smooth 0 AT LEVEL 0 3 UP: Norm after smooth 0 AT LEVEL 0 2 UP: Norm before smooth 0 AT LEVEL 0 2 UP: Norm after smooth 0 AT LEVEL 0 1 UP: Norm before smooth 0 AT LEVEL 0 1 UP: Norm after smooth 0 AT LEVEL 0 0 UP: Norm before smooth 0 AT LEVEL 0 0 UP: Norm after smooth 0 MLMG: Iteration 11 Fine resid/bnorm = 0 MLMG: Final Iter. 11 resid, resid/bnorm = 0, 0 MLMG: Timers: Solve = 2.045967801 Iter = 2.037279753 Bottom = 0.003868574

After projection:

  • On lev 0 max(abs(rhs)) = 0

Projection::level_project(): lev: 0, time: 2.065374957 NavierStokes::advance(): after velocity update max(abs(u/v)) = 0 0 max(abs(gpx/gpy/p)) = 0 0 0 [Level 0 step 6] Advanced 131072 cells

[STEP 6] Coarse TimeStep time: 2.447419548 [STEP 6] FAB kilobyte spread across MPI nodes: [226900 ... 226900]

STEP = 6 TIME = 0.0004388865319 DT = 9.161053351e-05

PLOTFILE: file = plt00006 Write plotfile time = 0.066868738 seconds

NavierStokesBase::estTimeStep() failed to provide a good timestep (probably because initial velocity field is zero with no external forcing). Use ns.init_dt to provide a reasonable timestep on coarsest level. Note that ns.init_shrink will be applied to init_dt. amrex::Abort::0:: !!! SIGABRT See Backtrace.0 file for details

I don't know if there's anything obvious that stands out here, the initial iteration of the MG solver doesn't look unusual compared to the other time steps, but starts blowing up after iteration 1 or 2 whereas it converges in the previous time step.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/IAMR/issues/46#issuecomment-652565635, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YVV2T7G3D76IJVJPJLRZN2RTANCNFSM4OE3BKNA .

-- Ann Almgren Senior Scientist; CCSE Group Lead

dcoveney commented 4 years ago

Sure, I can send you my version of the source and inputs files - should I post them here or send an email with file share link?

asalmgren commented 4 years ago

Email would be better

Sent from my iPhone

On Jul 1, 2020, at 11:16 AM, dcoveney notifications@github.com wrote:



Sure, I can send you my version of the source and inputs files - should I post them here or send an email with file share link?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/IAMR/issues/46#issuecomment-652573295, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YR53ZRJCMYVMU3AIYLRZN4P5ANCNFSM4OE3BKNA .

asalmgren commented 3 years ago

Closing this issue now since I think there's been a lot of offline discussion since this was opened ...