trixi-framework / Trixi.jl

Trixi.jl: Adaptive high-order numerical simulations of conservation laws in Julia
https://trixi-framework.github.io/Trixi.jl
MIT License
505 stars 98 forks source link

Restart does not work with TreeMesh + AMR + MPI #1912

Closed benegee closed 1 month ago

benegee commented 2 months ago

Extend examples/tree_2d_dgsem/elixir_advection_extended.jl by an AMR callback like

amr_controller = ControllerThreeLevel(semi, IndicatorMax(semi, variable = first),
                                      base_level = 0,
                                      med_level = 1, med_threshold = 0.8,
                                      max_level = 2, max_threshold = 1.2)
amr_callback = AMRCallback(semi, amr_controller,
                           interval = 5,
                           adapt_initial_condition = false,
                           adapt_initial_condition_only_refine = true)

(possibly also tspan and intervals have to be changed to make sure the expected restart file is indeed written)

Then execute examples/tree_2d_dgsem/elixir_advection_restart.jl in parallel, e.g. via

 ~/bin/mpiexecjl -n 2 julia --project=. -e 'using Trixi;  trixi_include("../examples/tree_2d_dgsem/elixir_advection_restart.jl")'

The following error occurs:

ERROR: LoadError: AssertionError: New length would exceed capacity
Stacktrace:
  [1] insert!(c::Trixi.ParallelTree{2}, position::Int64, count::Int64)
    @ Trixi ~/trixi/Trixi.jl/src/auxiliary/containers.jl:187
  [2] refine_unbalanced!(t::Trixi.ParallelTree{2}, cell_ids::Vector{Int64}, sorted_unique_cell_ids::Vector{Int64})
    @ Trixi ~/trixi/Trixi.jl/src/meshes/abstract_tree.jl:327

This happens because load_mesh_parallel does not read the capacity attribute and instead sets capacity to the number of actual cells (n_cells_max is set to 0 by default)