Mesh refinement in RZ geometry

[spadova-a]

Hello there,

I tried to run simulation in RZ geometry with mesh refinement, but the code is not able to build my computational domain (I am getting a "Segfault"), although the simulation runs fine without the MR. Can you please take a look at my input file and suggest a solution of the problem? Thank you very much.

inputRZ_MR.txt

[EZoni]

Hi @spadova-a, thank you for opening the issue.

I see in your input file that the field boundary conditions (boundary.field_lo and boundary.field_hi) are set to damped at zmin and zmax.

However, damped is a type of boundary conditions only available for the spectral solver, namely when algo.maxwell_solver = psatd, while in your case the Maxwell solver is set to algo.maxwell_solver = Yee and is not spectral.

We recommend trying another type of boundary conditions and checking if the simulation runs correctly. Please let us know if this works.

[spadova-a]

Hi EZoni,

Thanks for your answer, I corrected the boundary conditions to silver mueller (with damped type the simulation does not run with or without the MR, not really sure why I tried to use this). However, I'm still not able to run my RZ simulation with mesh refinement.

inputRZ_MR2.txt

[EZoni]

Thank you for updating and sharing the input script, @spadova-a. I think I can reproduce the issue and will report on what I find asap.

[EZoni]

I'm posting here, but it's more for internal debugging and discussion. Will edit and update this same comment as more information comes in.

I think the segfault is caused by the fact that in FullDiagnostics::InitializeFieldFunctorsRZopenPMD we try to get the pointer to Efield_aux[lev], for lev>0, before it is allocated.

By adding a few print statements here and there, this seems to be the backtrace of function calls:

• InitFromScratch()
  • AmrCore::InitFromScratch()
  • MakeNewLevelFromScratch(lev=0, ...)
    • AllocLevelData(lev=0, ...)
    • AllocLevelMFs(lev=0, ...)
      • AliasInitMultiFab(Efield_aux[lev=0], ...)
• InitDiagnostics()
  • multi_diags->InitData()
  • InitializeFieldFunctors(lev=0)
    • FullDiagnostics::InitializeFieldFunctorsRZopenPMD(lev=0)
  • InitializeFieldFunctors(lev=1)
    • FullDiagnostics::InitializeFieldFunctorsRZopenPMD(lev=1) (here we try to get the pointer to Efield_aux[lev=1] before it is allocated)

Somehow we don't get to the point where we allocate Efield_aux[lev=1] via AllocLevelMFs(lev=1, ...). This should be done automatically via AmrMesh::MakeNewGrids which is supposed to call MakeNewLevelFromScratch on each level.

This is what happens when when the code calls AmrMesh::MakeNewGrids(time):

• Call MakeNewLevelFromScratch(lev=0, ...)
• Call MakeNewGrids(finest_level=0, time=0, new_finest, ...)
• At this point: finest_level = 0 and new_finest = 0
• Call MakeNewGrids(0, time=0, new_finest, ...) (because iterate_on_new_grids = 1)
• At this point: finest_level = 0 and new_finest = 0

For some reason it does not manage to construct the grids of level 1.

[EZoni]

@WeiqunZhang @atmyers Do you have any idea about why this could be happening, what I described in the comment above: https://github.com/ECP-WarpX/WarpX/issues/3716#issuecomment-1482092913?

[EZoni]

@spadova-a After looking more into this, we figured out that the reason why AMReX is not creating the level 1 grids is that the mesh refinement patch set in the input is actually outside the computational domain:

geometry.prob_lo = 0 0
geometry.prob_hi = 3.4e-05 5.44e-05
warpx.fine_tag_lo = 0 209.8e-6
warpx.fine_tag_hi = 200e-9 211.2e-6

The z domain for the fine patch is [2.098e-4, 2.112e-4], which is outside the computational z domain [0, 5.44e-5].

If this setup was not intended, you should be able to see the level 1 grids created by AMReX once you fix the input file.

If this setup was intended, because you want the MR region to be added once the moving window gets there, the issue might be addressed by fixing some of the for loops on the MR levels that lead to the segfault, making sure that they use the current finest_level extracted from WarpX rather than the max_level set in the input, to be more robust with respect to situations where the two quantities don't match.

I think we should add this fix anyways, but I would be interested in knowing if your input setup was intended as is.

[spadova-a]

Hello @EZoni, yes this is my intended setup - I study electron injection caused by a presence of nanoparticle, so I wanted the reformed region to be only around the nanoparticle.

[EZoni]

@spadova-a Thank you for confirming. I think #3798 should fix the initial segfault that you encountered. It would be good if you could try rerunning your test and confirm that. You might encounter additional issues later in time, when the MR grids are actually created in the simulation, in which case please feel free to report them here, too.

[ax3l]

PR #3798 merged - @spadova-a feel free to try again! :)