Mesh refinement near domain edge causes seg fault

[rybchuk]

I am in the process of setting up an ABL + wave simulation where I would like to have refinement near the air-water interface. I'd like this refinement zone to run from x=0 to x=Lx and y=0 and y=Lx. I encountered a seg fault when I tried to run the full simulation.

It turns out this seg fault is coming from the mesh refinement, and it has nothing to do with the offshore physics. I ran a small test case with a pure atmospheric simulation. If I keep the refinement zone at least 150 m away from the boundary, the simulation works. If I move it within 135 m of the inflow boundary, it breaks. My hunch is that this has something to do with the little grids that get split up across the cores. This issue also seems related to, but distinct from #735.

Grid summary:
  Level 0   256 grids  262144 cells  100 % of domain
            smallest grid: 16 x 8 x 8  biggest grid: 16 x 8 x 8
  Level 1   294 grids  602112 cells  28.7109375 % of domain
            smallest grid: 16 x 16 x 8  biggest grid: 16 x 16 x 8

Shear Stress model: moeng
Heat Flux model: moeng
Creating MomentumSource instance: BoussinesqBuoyancy
Creating MomentumSource instance: CoriolisForcing
Geostrophic forcing: Coriolis factor = 0.0001458423163
Creating MomentumSource instance: GeostrophicForcing
srun: error: x1000c0s0b1n0: tasks 149,176: Segmentation fault (core dumped)

I've attached the input files for the working and broken simulations, the associated log files, as well as the job submission script. Some other details:

• I compiled on Kestrel today using the latest version of the code
• I tested both Geostrophic forcing and ABL forcing, but both seg fault
• I ran with 500+ cores and saw the seg fault still happen
• I also ran with 128 cores and saw the deg fault happen

If anyone has suggestions on how to debug this or why it may be happening, I'd greatly appreciate it, thanks!

mesh_debugging.zip

[rthedin]

I've had similar issues in the past week or so, until I realized it was related to Kestrel. My cases were working on Eagle. It turned out, with the environment changes, I needed to compile the tools using a different set of modules. That has fixed my issue last week. Regarding your case, I just tested the broken.i input on Kestrel on my newest binary and it works fine. I suggest you try on Eagle (or other system) to rule that out, as I believe you might be having the same compilation issue I had.

[marchdf]

@rybchuk did @rthedin 's solution solve the problem?

[rybchuk]

Running on Eagle did indeed solve the problem!