search

AMReX-Combustion / PeleC

An AMR code for compressible reacting flow simulations
https://amrex-combustion.github.io/PeleC
Other
162 stars 73 forks source link

Encounter "IndexSpaceImp: domain is not coarsenable" problems when restarting and increasing the AMR level #772

Closed EarlFan closed 7 months ago

EarlFan commented 7 months ago

Hello,

When I am trying to restart from a chk file with increasing the AMR maximum level from 3 to 5, or from 4 to 5, PeleC fails and reports

Initializing AMReX (23.12-24-g75571e2dcbf2)...
MPI initialized with 10 MPI processes
MPI initialized with thread support level 0
amrex::Abort::1::IndexSpaceImp: domain is not coarsenable at level 5 !!!
SIGABRT
amrex::Abort::2::IndexSpaceImp: domain is not coarsenable at level 5 !!!
SIGABRT
amrex::Abort::3::IndexSpaceImp: domain is not coarsenable at level 5 !!!
SIGABRT
amrex::Abort::4::IndexSpaceImp: domain is not coarsenable at level 5 !!!

NOTING that I succeed restarting the simulation when increasing the AMR maximum level from 0 to 3, or from 3 to 4.

After some googlings, I find the reported error is related to a EB function. https://amrex-codes.github.io/amrex/doxygen/AMReX__EB2__IndexSpaceI_8H_source.html. However, EB is not used in the simulation.

Please help me with the problem and any help will be appreciated.

baperry2 commented 7 months ago

I haven't seen something like this before for a simulation without EB. Can you provide more details on the case you are running, or reproduce this issue using one of the standard RegTests?

PeleC is always compiled with AMReX EB support active, but most EB code paths should only come into play if there is EB in your domain.

EarlFan commented 7 months ago

Thanks for your reply!

I encounter this problem when simulating a simple rectangular channel detonation case, and I didn't apply EB in the .inp file which is shown as below.

# ------------------  INPUTS TO MAIN PROGRAM  -------------------
max_step = 10000
stop_time =  8.e-6 #final time is 0.2*L*sqrt(rhoL/pL)

# delete information for the z-axis grids - fane
# PROBLEM SIZE & GEOMETRY
geometry.is_periodic = 0  1
geometry.coord_sys   = 0  # 0 => cart, 1 => RZ  2=>spherical
geometry.prob_lo     = 0.0  0.0
geometry.prob_hi     = 1.6 0.04
amr.n_cell           =  640 16

# delete information for the z-axis boundaries - fane
# >>>>>>>>>>>>>  BC KEYWORDS <<<<<<<<<<<<<<<<<<<<<<
# Interior, UserBC, Symmetry, SlipWall, NoSlipWall
# >>>>>>>>>>>>>  BC KEYWORDS <<<<<<<<<<<<<<<<<<<<<<
pelec.lo_bc       =  "SlipWall"  "Interior"
pelec.hi_bc       =  "FOExtrap"  "Interior"

# WHICH PHYSICS
# pelec.do_hydro = 1
# pelec.do_mol = 0
# pelec.diffuse_vel = 0
# pelec.diffuse_temp = 0
# pelec.diffuse_spec = 0
# pelec.do_react = 1
# pelec.diffuse_enth = 0

pelec.do_hydro = 1
pelec.do_mol   = 0
pelec.do_react = 1
pelec.diffuse_vel = 0
pelec.diffuse_temp = 0
pelec.diffuse_spec = 0
pelec.diffuse_enth = 0
pelec.chem_integrator = ReactorArkode # ReactorCvode # "ReactorCvode" "ReactorRK64" "ReactorArkode"
cvode.solve_type = "dense_direct" #"precGMRES" # "dense_direct" # "precGMRES" # "GMRES"
cvode.precond_type = "dense_simplified_AJacobian"
ode.rtol = 1e-6
ode.atol = 1e-6
ode.max_nls_iters = 10000
ode.analytical_jacobian = 1
ode.clean_init_massfrac = 1
pelec.allow_negative_energy = 1
pelec.diffuse_temp=0
pelec.diffuse_enth=0
pelec.diffuse_spec=0
pelec.diffuse_vel=0
pelec.sdc_iters = 2
pelec.flame_trac_name = P

# TIME STEP CONTROL
# pelec.fixed_dt = 1e-9
# pelec.sdc_iters = 1

pelec.dt_cutoff      = 5.e-20  # level 0 timestep below which we halt
pelec.cfl            = 0.3     # cfl number for hyperbolic system
pelec.init_shrink    = 0.2     # scale back initial timestep
pelec.change_max     = 1.2     # scale back initial timestep
# pelec.max_dt         = 1e-7    # the largest valid timestep
#pelec.change_max     = 1.05    # scale back initial timestep

# DIAGNOSTICS & VERBOSITY
pelec.sum_interval   = 1       # timesteps between computing mass
pelec.v              = 1       # verbosity in PeleC.cpp
amr.v                = 1       # verbosity in Amr.cpp
amr.data_log         = datlog

# REFINEMENT / REGRIDDING
amr.max_level       = 4      # maximum level number allowed
amr.ref_ratio       = 2 2 2 2 2 2 # refinement ratio
amr.regrid_int      = 2 2 2 2 2 2 # how often to regrid
# amr.n_error_buf     = 0 0 0 0 # number of buffer cells in error est
amr.blocking_factor = 8       # block factor in grid generation
amr.max_grid_size   = 64
amr.grid_eff        = 0.99
amr.n_error_buf     = 4 4 4 4 4 4 # number of buffer cells in error est

# CHECKPOINT FILES
amr.checkpoint_files_output = 1
amr.check_file      = chk        # root name of checkpoint file
# amr.check_int       = -1        # number of timesteps between checkpoints
amr.check_per       = 1e-7        # number of timesteps between checkpoints

# PLOTFILES
amr.plot_files_output = 1
amr.file_name_digits  = 7
amr.plot_file       = plt        # root name of plotfile
# amr.plot_int        = 100        # number of timesteps between plotfiles
amr.plot_per        = 1e-7       # number of timesteps between plotfiles
amr.plot_vars = ALL
amr.derive_plot_vars = ALL
# amr.derive_plot_vars = MachNumber x_velocity y_velocity z_velocity pressure

# **** start PROBLEM PARAMETERS ****

# **** mixture setups 
# unshocked states
prob.T_unshocked = 298.
prob.p_unshocked = 1013250.
prob.u_unshocked = 0.
prob.rhos_unshocked = 0.001267733179918895 0.  

# shocked states
prob.T_shocked = 2086.0
prob.p_shocked = 7092750.
# prob.u_shocked = 60000.
prob.u_shocked = 0.
prob.rhos_shocked = 0.001267733179918895 0. 

# bubble states
prob.T_bubble = 897.0
prob.p_bubble = 3049950.
prob.u_bubble = 0.
prob.rhos_bubble = 0.001267733179918895 0.  

# **** geometric setups 
prob.x_shock_ini = 0.15
prob.x_center = 14e-2
prob.y_center = 2e-2
prob.bubble_l = 1e-2
prob.bubble_w = 2e-2

# *** particles
# prob.part_num = 200

# **** end PROBLEM PARAMETERS ****

#prob.p_l = 1e7
#prob.rho_l = 9.6e-4
#prob.p_r = 1e6
#prob.rho_r = 1.2e-4
#prob.angle = 0.0 # 30 to 0 - fane
#prob.left_gas = N2
#prob.right_gas = XE

# Problem setup
# pelec.eb_boundary_T = 300.
# pelec.eb_isothermal = 0

# TAGGING
#tagging.denerr = 1e20
#tagging.dengrad = 4e-5
#tagging.max_denerr_lev = 3
#tagging.max_dengrad_lev = 3
#tagging.tempgrad = 50.0
#tagging.max_tempgrad_lev = 3

tagging.max_pressgrad_lev = 5
tagging.pressgrad = 4e6
tagging.dengrad = 1e-2

# PARTICLES / SPRAY
pelec.do_spray_particles = 0

extern.new_Jacobian_each_cell = 0

Your reply reminds me to check whether EB-related functions are called even I didn't use eb2 in the .inp file. As I use the GNUmakefile and make to compile the PeleC***.exe, I search the GNUmakefile and find the USE_EB option in the PeleC/Exec/Make.PeleC

PELE_HOME        ?= ../
PELE_PHYSICS_HOME ?= $(abspath $(lastword $(PELE_HOME)/Submodules/PelePhysics))
AMREX_HOME        ?= $(abspath $(lastword $(PELE_PHYSICS_HOME)/Submodules/amrex))
SUNDIALS_HOME     ?= $(abspath $(lastword $(PELE_PHYSICS_HOME)/Submodules/sundials))

EBASE = PeleC

# EXTERN_CORE is simply a list of the directories we use in Eos, Reactions, Transport and Util
# this list will be searched for runtime parameters
EXTERN_CORE ?=

#Turn off all fortran in amrex
ifneq ($(USE_MASA), TRUE)
  BL_NO_FORT = TRUE
endif

USE_EB = TRUE
USE_SUNDIALS = TRUE

When change the USE_EB to FALSE in the PeleC/Exec/Make.PeleC, PeleC cannot be compiled and reports the following errors relevant to EB:

g++ -MMD -MP  -Werror=return-type -g -O0 -ggdb -ftrapv -Wall -Wextra -Wlogical-op -Wfloat-conversion -Wnull-dereference -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wmissing-include-dirs -Wpedantic -Wshadow -Woverloaded-virtual -Wnon-virtual-dtor -std=c++17  -pthread    -DAMREX_DEBUG -DBL_NO_FORT -DAMREX_GPU_MAX_THREADS=0 -DBL_SPACEDIM=2 -DAMREX_SPACEDIM=2 -DBL_FORT_USE_UNDERSCORE -DAMREX_FORT_USE_UNDERSCORE -DBL_Linux -DAMREX_Linux -DBL_USE_ASSERTION -DAMREX_USE_ASSERTION -DAMREX_PARTICLES -DCRSEGRNDOMP -DUSE_FUEGO_EOS -DUSE_SIMPLE_TRANSPORT -DPELE_USE_SPRAY -DSPRAY_FUEL_NUM=1 -DAMREX_USE_SUNDIALS -Itmp_build_dir/s/2d.gnu.DEBUG.EXE -I. -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Extern/SUNDIALS -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility/ -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Eos/ -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Reactions -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/ThirdParty/INSTALL/gnu.DEBUG/lib/../include -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Transport -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility/TurbInflow -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility/PltFileManager -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility/Diagnostics -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility/Filter -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Eos -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Reactions -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Transport -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Spray -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Spray/Distribution -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Spray/BreakupSplash -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Base -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Base/Parser -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Amr -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Boundary -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/AmrCore -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Particle -I. -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility//TurbInflow -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility//PltFileManager -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility//Diagnostics -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Utility//Filter -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Eos/ -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Reactions -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Mechanisms/Conaire -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Transport -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Spray -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Spray/Distribution -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Source/Spray/BreakupSplash -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Base -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Amr -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Boundary -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/AmrCore -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/Particle -I../../../Source -I../../../Source/Params/param_includes -I/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Tools/C_scripts   -c /home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp -o tmp_build_dir/o/2d.gnu.DEBUG.EXE/AMReX_EBMFInterpolater.o
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp: In member function ‘virtual void amrex::EBMFCellConsLinInterp::interp(const amrex::MultiFab&, int, amrex::MultiFab&, int, int, const amrex::IntVect&, const amrex::Geometry&, const amrex::Geometry&, const amrex::Box&, const amrex::IntVect&, const amrex::Vector<amrex::BCRec>&, int)’:
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:19:44: error: ‘EBFArrayBoxFactory’ does not name a type; did you mean ‘FArrayBoxFactory’?
   19 |     const auto *const ffact = dynamic_cast<EBFArrayBoxFactory const*>(&finemf.Factory());
      |                                            ^~~~~~~~~~~~~~~~~~
      |                                            FArrayBoxFactory
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:19:63: error: expected ‘>’ before ‘const’
   19 |     const auto *const ffact = dynamic_cast<EBFArrayBoxFactory const*>(&finemf.Factory());
      |                                                               ^~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:19:63: error: expected ‘(’ before ‘const’
   19 |     const auto *const ffact = dynamic_cast<EBFArrayBoxFactory const*>(&finemf.Factory());
      |                                                               ^~~~~
      |                                                               (
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:19:63: error: expected primary-expression before ‘const’
   19 |     const auto *const ffact = dynamic_cast<EBFArrayBoxFactory const*>(&finemf.Factory());
      |                                                               ^~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:19:62: error: expected ‘)’ before ‘const’
   19 |     const auto *const ffact = dynamic_cast<EBFArrayBoxFactory const*>(&finemf.Factory());
      |                                                              ^~~~~~
      |                                                              )
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:20:44: error: ‘EBFArrayBoxFactory’ does not name a type; did you mean ‘FArrayBoxFactory’?
   20 |     const auto *const cfact = dynamic_cast<EBFArrayBoxFactory const*>(&crsemf.Factory());
      |                                            ^~~~~~~~~~~~~~~~~~
      |                                            FArrayBoxFactory
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:20:63: error: expected ‘>’ before ‘const’
   20 |     const auto *const cfact = dynamic_cast<EBFArrayBoxFactory const*>(&crsemf.Factory());
      |                                                               ^~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:20:63: error: expected ‘(’ before ‘const’
   20 |     const auto *const cfact = dynamic_cast<EBFArrayBoxFactory const*>(&crsemf.Factory());
      |                                                               ^~~~~
      |                                                               (
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:20:63: error: expected primary-expression before ‘const’
   20 |     const auto *const cfact = dynamic_cast<EBFArrayBoxFactory const*>(&crsemf.Factory());
      |                                                               ^~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:20:62: error: expected ‘)’ before ‘const’
   20 |     const auto *const cfact = dynamic_cast<EBFArrayBoxFactory const*>(&crsemf.Factory());
      |                                                              ^~~~~~
      |                                                              )
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:59:19: error: ‘EBCellFlagFab’ does not name a type
   59 |             const EBCellFlagFab& crse_flag_fab = cflags[mfi];
      |                   ^~~~~~~~~~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:61:23: error: ‘EBCellFlagFab’ does not name a type
   61 |                 const EBCellFlagFab& fine_flag_fab = fflags[mfi];
      |                       ^~~~~~~~~~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:62:39: error: ‘fine_flag_fab’ was not declared in this scope
   62 |                 const FabType ftype = fine_flag_fab.getType(target_fine_region);
      |                                       ^~~~~~~~~~~~~
/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Src/EB/AMReX_EBMFInterpolater.cpp:63:39: error: ‘crse_flag_fab’ was not declared in this scope
   63 |                 const FabType ctype = crse_flag_fab.getType(crse_bx);
      |                                       ^~~~~~~~~~~~~
make: *** [/home/fane/PeleC/PeleC/Submodules/PelePhysics/Submodules/amrex/Tools/GNUMake/Make.rules:262: tmp_build_dir/o/2d.gnu.DEBUG.EXE/AMReX_EBMFInterpolater.o] Error 1

I guess that for somehow, EB models have to be included to compile AMReX, even though the EB model is not explicitly called in the PeleC case.

marchdf commented 7 months ago

We require that USE_EB=TRUE when building PeleC. We include EB functionality by default, even if it not used (this removes a whole bunch of duplicate code and the use of ifdefs throughout the code and generally makes things easier to debug/test).

We would be happy to take a closer look if you provide us with your problem files. The easiest is to point us to a branch.

EarlFan commented 7 months ago

Hello,

I have prepared the PeleC case files and Pelephysics mechanism files for the problem debugging, and I'd like to provide the files in a new branch following the Development model instructions. However, I cannot push the new branch to PeleC due to Permission problems.

$ git push -u origin fane
ERROR: Permission to AMReX-Combustion/PeleC.git denied to EarlFan.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Can anyone help me with the permission of pushing this new branch? Thanks!

marchdf commented 7 months ago

you won't have permissions to do that. You should create a fork and push there. I updated the instructions: https://github.com/AMReX-Combustion/PeleC/pull/776

EarlFan commented 7 months ago

Please find the PeleC case files and Pelephysics mechanism files for the issue debugging. https://github.com/EarlFan/PeleC/commit/029509ddfaf3ddd0a75dc5ffc8b46b6f9359eab5

marchdf commented 7 months ago

Ok I can confirm I see the same thing. Thank you for sharing the files and creating the readme

marchdf commented 7 months ago

I have 2 solutions for you:

  1. start you first run (the coarse, single level one with more cells, e.g. 1280 x 32 cells) and then you can restart from that with up to 5 levels.
  2. With your current setup, you can force things to work by doing adding eb2.max_level_generation=5

The issue is that the coarse grid is just too coarse and can't restart with 5 levels. Let me know if these don't work for you.

EarlFan commented 7 months ago

Thanks for your help! It works well.