AMReX-Astro / Castro

Castro (Compressible Astrophysics): An adaptive mesh, astrophysical compressible (radiation-, magneto-) hydrodynamics simulation code for massively parallel CPU and GPU architectures.
http://amrex-astro.github.io/Castro
Other
299 stars 97 forks source link

make_radial_gravity errors after retry w/ subcycling disabled #2820

Closed zingale closed 5 months ago

zingale commented 5 months ago

When running with AMR with no subcycling and monopole gravity, if we trigger a retry, then we get errors in the make_radial_gravity of the form:

 Level / Time in make_radial_gravity is: 0 0.00315
 but old / new time      are: 0.00165 0.002025
amrex::Abort::14::Problem in Gravity::make_radial_gravity !!!
SIGABRT
 Level / Time in make_radial_gravity is: 0 0.00315
 but old / new time      are: 0.00165 0.002025
amrex::Abort::13::Problem in Gravity::make_radial_gravity !!!
SIGABRT

I've seen this with the massive star problem but I can also reproduce with DustCollapse (see below)

zingale commented 5 months ago

reproducer:

in DustCollapse:

make DIM=3 -j 20

then use the attached inputs

inputs_3d_monopole_regtest.txt

and run as:

mpiexec -n 20 ./Castro3d.gnu.MPI.ex inputs_3d_monopole_regtest  max_step=20
zingale commented 5 months ago

note that the reproducer sets a fixed_dt to force problems and a retry.

zingale commented 5 months ago

this definitely seems to be related to doing amr.subcycling_mode=None

zingale commented 5 months ago

here's the issue. When we reject a step and retry w/o subcycling, it seems that we are not resetting LevelData[level]->get_state_data(State_Type).curTime(); on the finer levels.

Then when we enter into get_new_grav_vector(), we have a mismatch between the time that is passed in on level 1 and the time that is inferred from State_Type.

So I think in the reset for retry, we need to update all of the levels above us as well if we are not subcycling.