search

Exawind / amr-wind

AMReX-based structured wind solver
https://exawind.github.io/amr-wind
Other
110 stars 82 forks source link

Box refinement bounds are being applied to the wrong mesh level #1317

Closed justhawk98 closed 4 hours ago

justhawk98 commented 4 hours ago

Bug description

This might be user error but it seems that when I give AMR-Wind the bounds for box mesh refinement, the box bounds are being assigned to the level below what it should be. So for example, the box that should contain cells for AMR level 5 contains level 6 cells, and the box that should define the bounds for level 6 cells contain level 7 cells...etc.

There is an easy workaround as a user, I can just change the bounds for the level 6 box to define where I want my level 7 cells and so on, but I thought I should report this behavior.

Steps to reproduce

Here is the input file I was using when I first discovered this behavior: 5mwTurbine.inp_old.txt

Here are the relevant tagging sections:

tagging.labels = t2z0 t2z1 t2z2 t2z3 t2z4 t2z5 t2z6 t2z7 z0 z1 z2 z3 z4 z5 z6 z7

tagging.t2z0.type = GeometryRefinement tagging.t2z0.shapes = t2z0 tagging.t2z0.level = 0 tagging.t2z0.t2z0.type = box tagging.t2z0.t2z0.origin = 833.0 932.0 900.0 tagging.t2z0.t2z0.xaxis = 1063.0 0.0 0.0 tagging.t2z0.t2z0.yaxis = 0.0 720.0 0.0 tagging.t2z0.t2z0.zaxis = 0.0 0.0 720.0

tagging.t2z1.type = GeometryRefinement tagging.t2z1.shapes = t2z1 tagging.t2z1.level = 1 tagging.t2z1.t2z1.type = box tagging.t2z1.t2z1.origin = 905.0 1004.0 972.0 tagging.t2z1.t2z1.xaxis = 991.0 0.0 0.0 tagging.t2z1.t2z1.yaxis = 0.0 576.0 0.0 tagging.t2z1.t2z1.zaxis = 0.0 0.0 576.0

tagging.t2z2.type = GeometryRefinement tagging.t2z2.shapes = t2z2 tagging.t2z2.level = 2 tagging.t2z2.t2z2.type = box tagging.t2z2.t2z2.origin = 962.0 1061.0 1030.0 tagging.t2z2.t2z2.xaxis = 934.0 0.0 0.0 tagging.t2z2.t2z2.yaxis = 0.0 461.0 0.0 tagging.t2z2.t2z2.zaxis = 0.0 0.0 461.0

tagging.t2z3.type = GeometryRefinement tagging.t2z3.shapes = t2z3 tagging.t2z3.level = 3 tagging.t2z3.t2z3.type = box tagging.t2z3.t2z3.origin = 1008.0 1107.0 1076.0 tagging.t2z3.t2z3.xaxis = 888.0 0.0 0.0 tagging.t2z3.t2z3.yaxis = 0.0 369.0 0.0 tagging.t2z3.t2z3.zaxis = 0.0 0.0 369.0

tagging.t2z4.type = GeometryRefinement tagging.t2z4.shapes = t2z4 tagging.t2z4.level = 4 tagging.t2z4.t2z4.type = box tagging.t2z4.t2z4.origin = 1045.0 1144.0 1112.0 tagging.t2z4.t2z4.xaxis = 600.0 0.0 0.0 tagging.t2z4.t2z4.yaxis = 0.0 295.0 0.0 tagging.t2z4.t2z4.zaxis = 0.0 0.0 295.0

tagging.t2z5.type = GeometryRefinement tagging.t2z5.shapes = t2z5 tagging.t2z5.level = 5 tagging.t2z5.t2z5.type = box tagging.t2z5.t2z5.origin = 1075.0 1174.0 1142.0 tagging.t2z5.t2z5.xaxis = 437.0 0.0 0.0 tagging.t2z5.t2z5.yaxis = 0.0 236.0 0.0 tagging.t2z5.t2z5.zaxis = 0.0 0.0 236.0

tagging.t2z6.type = GeometryRefinement tagging.t2z6.shapes = t2z6 tagging.t2z6.level = 6 tagging.t2z6.t2z6.type = box tagging.t2z6.t2z6.origin = 1116.0 1197.0 1166.0 tagging.t2z6.t2z6.xaxis = 36.0 0.0 0.0 tagging.t2z6.t2z6.yaxis = 0.0 189.0 0.0 tagging.t2z6.t2z6.zaxis = 0.0 0.0 189.0

tagging.t2z7.type = GeometryRefinement tagging.t2z7.shapes = t2z7 tagging.t2z7.level = 7 tagging.t2z7.t2z7.type = box tagging.t2z7.t2z7.origin = 1125.0 1215.9 1184.4 tagging.t2z7.t2z7.xaxis = 18.0 0.0 0.0 tagging.t2z7.t2z7.yaxis = 0.0 151.2 0.0 tagging.t2z7.t2z7.zaxis = 0.0 0.0 151.2

tagging.z0.type = GeometryRefinement tagging.z0.shapes = z0 tagging.z0.level = 0 tagging.z0.z0.type = box tagging.z0.z0.origin = 76.0 900.0 900.0 tagging.z0.z0.xaxis = 1820.0 0.0 0.0 tagging.z0.z0.yaxis = 0.0 720.0 0.0 tagging.z0.z0.zaxis = 0.0 0.0 720.0

tagging.z1.type = GeometryRefinement tagging.z1.shapes = z1 tagging.z1.level = 1 tagging.z1.z1.type = box tagging.z1.z1.origin = 148.0 972.0 972.0 tagging.z1.z1.xaxis = 1748.0 0.0 0.0 tagging.z1.z1.yaxis = 0.0 576.0 0.0 tagging.z1.z1.zaxis = 0.0 0.0 576.0

tagging.z2.type = GeometryRefinement tagging.z2.shapes = z2 tagging.z2.level = 2 tagging.z2.z2.type = box tagging.z2.z2.origin = 206.0 1030.0 1030.0 tagging.z2.z2.xaxis = 1690.0 0.0 0.0 tagging.z2.z2.yaxis = 0.0 461.0 0.0 tagging.z2.z2.zaxis = 0.0 0.0 461.0

tagging.z3.type = GeometryRefinement tagging.z3.shapes = z3 tagging.z3.level = 3 tagging.z3.z3.type = box tagging.z3.z3.origin = 252.0 1076.0 1076.0 tagging.z3.z3.xaxis = 1644.0 0.0 0.0 tagging.z3.z3.yaxis = 0.0 369.0 0.0 tagging.z3.z3.zaxis = 0.0 0.0 369.0

tagging.z4.type = GeometryRefinement tagging.z4.shapes = z4 tagging.z4.level = 4 tagging.z4.z4.type = box tagging.z4.z4.origin = 288.0 1112.0 1112.0 tagging.z4.z4.xaxis = 1608.0 0.0 0.0 tagging.z4.z4.yaxis = 0.0 295.0 0.0 tagging.z4.z4.zaxis = 0.0 0.0 295.0

tagging.z5.type = GeometryRefinement tagging.z5.shapes = z5 tagging.z5.level = 5 tagging.z5.z5.type = box tagging.z5.z5.origin = 318.0 1142.0 1142.0 tagging.z5.z5.xaxis = 819.0 0.0 0.0 tagging.z5.z5.yaxis = 0.0 236.0 0.0 tagging.z5.z5.zaxis = 0.0 0.0 236.0

tagging.z6.type = GeometryRefinement tagging.z6.shapes = z6 tagging.z6.level = 6 tagging.z6.z6.type = box tagging.z6.z6.origin = 355.0 1166.0 1166.0 tagging.z6.z6.xaxis = 36.0 0.0 0.0 tagging.z6.z6.yaxis = 0.0 189.0 0.0 tagging.z6.z6.zaxis = 0.0 0.0 189.0

tagging.z7.type = GeometryRefinement tagging.z7.shapes = z7 tagging.z7.level = 7 tagging.z7.z7.type = box tagging.z7.z7.origin = 364.0 1184.4 1184.4 tagging.z7.z7.xaxis = 18.0 0.0 0.0 tagging.z7.z7.yaxis = 0.0 151.2 0.0 tagging.z7.z7.zaxis = 0.0 0.0 151.2

This is a simulation of a two turbine wind farm, and each turbine has successive levels of mesh refinement around it. I realized the mesh boxes were not conforming to the bounds set by the tagging levels when I visualized the mesh in ParaView, see below:

whole_domain line

In the 2nd image, I am zoomed in to the refinement box surrounding the upstream turbine. I added a line to show the box's measurements. You can see that the finest box (Level 7) is 36m wide in the x direction. The cells in that box are the correct length for level 7. At the coarsest level, there are 26 cells in the x-direction across a domain 1896m long. So the cells at the coarsest level are roughly 73m long. 73m*0.5^(7)=0.57m, which is the length I measured those cells to be in the finest box. This is what led me to believe that the box which should define level 6 contains level 7 cells. As you can see in the tagging.z6 section, this 36m box spanning from x=355 to x=391 should contain level 6 cells. Additionally, the box defined in tagging.z7 is seemingly absent.

After realizing this, it was a quick fix to just replace the bounds of each mesh box with the bounds of the level below it. This produced the mesh I wanted, seen below: corrected And here's the updated input file: 5mwTurbine.inp.txt

Is this just user error? I might just be misunderstanding the way refinement boxes are defined.

Steps to reproduce the behavior:

  1. Compiler used
    • [x ] GCC
    • [ ] LLVM
    • [ ] oneapi (Intel)
    • [ ] nvcc (NVIDIA)
    • [ ] rocm (AMD)
    • [x ] with MPI
    • [ ] other:
  2. Operating system
    • [ x] Linux
    • [ ] OSX
    • [ ] Windows
    • [ ] other (do tell ;)):
  3. Hardware:
    • [x ] CPU
    • [ ] GPU
  4. Machine details ():

BYU supercomputer

============================================================================== AMR-Wind (https://github.com/exawind/amr-wind)

AMR-Wind version :: v2.1.0-6-g80862cd4 AMR-Wind Git SHA :: 80862cd4dcb88d32f23b81d84a3d4911a8a4e590 AMReX version :: 24.03-36-g748f8dfea597

Exec. time :: Fri Oct 25 15:08:52 2024 Build time :: Jun 21 2024 09:30:00 C++ compiler :: GNU 14.1.0

MPI :: ON (Num. ranks = 128) GPU :: OFF OpenMP :: OFF

Enabled third-party libraries: OpenFAST 

       This software is released under the BSD 3-clause license.

See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details.

marchdf commented 4 hours ago

I think this is poor documentation on our part. We should fix that. But basically the "level" argument is the level on which the refinement is applied (i.e. the level below where the refinement happens). From the docs:

If level is provided and is greater than or equal to 0, then the refinement based on geometries defined for this section is only performed at that level.

I will admit that that could be improved ;)

mbkuhn commented 4 hours ago

Adding on to Marc's explanation above:

For your case, the reason you don't see refinement with the bounds specified with level = 7 is because the max_level of your input file is 7. Because the tagging arguments work the way they do (as Marc explained), you would see refinements at those bounds when amr.max_level is set to 8. The tagging at level 7 tells the code to refine at those locations, creating level 8 cells, but level 8 cells will not be created if the max_level is less than 8.

We'll try to add to the documentation soon to clarify this.

justhawk98 commented 4 hours ago

It's all making better sense now. Thanks you two!