octree support for C++ application codes?

[BenWibking]

Hi,

I have an unigrid finite volume application code developed in C++ that I want to port to an AMR framework, ideally AMReX. However, I would like to use octree AMR in order to allow for grid/block-level subcycling. Unfortunately, I cannot find support for octree-style AMR with C++ applications mentioned in the documentation. Is there support for this, or is this planned?

Thanks! Ben

[asalmgren]

Ben,

AMReX does not currently have traditional octree support, although we have started some efforts to support octree-style gridding.

I'm curious that you say you want octree for grid/block-level subcycling -- traditional octree does not subcycle. Can you clarify what you're wanting to do?

On Mon, Jun 29, 2020 at 8:47 PM Ben Wibking notifications@github.com wrote:

Hi,

I have an unigrid finite volume application code developed in C++ that I want to port to an AMR framework, ideally AMReX. However, I would like to use octree AMR in order to allow for grid/block-level subcycling. Unfortunately, I cannot find support for octree-style AMR with C++ applications mentioned in the documentation. Is there support for this, or is this planned?

Thanks! Ben

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/amrex/issues/1070, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YVMRK72EYR6WNXF3XLRZFN4DANCNFSM4OL2ATXA .

-- Ann Almgren Senior Scientist; CCSE Group Lead

[rmrsk]

In addition to only supporting square domains with factor 2 refinement, the bads news about an octree is that the fine level does not move "freely" inside the coarse level. You can still subcycle, but you are locked into having a buffer zone of half a leaf width.

There are tile-based refinement algorithms that relax the first two requirements, but they dont resolve the buffer issue.

Still, if your problems permit this decomposition and you are willing to eat the buffer penalty, then using an octree is effectively a matter of writing an algorithm that generates the boxes according to octree rules. You won't be able to index your AMR boxes with a tree, I think. Patch based formalisms have a more general way of doing that...

[BenWibking]

A few astrophysics codes have of late implemented grid-level subcycling with octree AMR. I suppose it doesn't have to be octree per se, but an AMR scheme that avoids having parent/child overlapping grids is necessary, as is usually the case with octree AMR.

This way, if there is, say, a localized hotspot, the fast sound speed in the hotspot only reduces the timestep on the grid it lives on, rather than affecting the timestep of all covering grids, including the root grid (and therefore it also reduces the global timestep). The efficiency gains on some problems appear to be quite significant (at least for astrophysical applications where the accuracy requirements can be somewhat relaxed). It sounds like the octree support in AMReX unfortunately may not support such a use-case, but please clarify if my understanding of your earlier comments is incorrect.

[asalmgren]

Ben,

Are you talking about codes that have no self-gravity solves? Otherwise I'm not sure at what time levels you would do the solves....

If everything is explicit one could certainly implement something like this in AMReX (it would take some modifications) but one would have to be careful. If you had two grids at the same spatial resolution but taking different time steps, one would need to make sure to correctly interpolate the ghost cell data in time ... that then becomes a very local procedure which would probably reduce the at-scale performance

So I can definitely see this if the fine grid with the subcycling does not touch other grids at the same resolution but otherwise I think it will pretty negatively impact the ghost cell filling performance.

Do you have a specialized use case or are you wanting a very general functionality?

On Wed, Jul 1, 2020 at 3:15 PM Ben Wibking notifications@github.com wrote:

A few astrophysics codes have of late implemented grid-level subcycling with octree AMR. I suppose it doesn't have to be octree per se, but an AMR scheme that avoids having parent/child overlapping grids is necessary, as is usually the case with octree AMR.

This way, if there is, say, a localized hotspot, the fast sound speed in the hotspot only reduces the timestep on the grid it lives on, rather than affecting the timestep of all covering grids, including the root grid (and therefore it also reduces the global timestep). The efficiency gains on some problems appear to be quite significant (at least for astrophysical applications where the accuracy requirements can be somewhat relaxed). It sounds like the octree support in AMReX unfortunately may not support such a use-case, but please clarify if my understanding of your earlier comments is incorrect.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/amrex/issues/1070#issuecomment-652672689, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YVTUDPU7DRBBCOGWDLRZOYP5ANCNFSM4OL2ATXA .

-- Ann Almgren Senior Scientist; CCSE Group Lead

[WeiqunZhang]

Ben,

Are you looking subcyling or octree? AMReX supports subcyling naturally. Most AMReX codes do subcyling. In fact, for amrex/Src/Amr based codes, one has to use a runtime parameter to turn off subcycling because it's on by default. If what you want is subcyling, AMReX supports it and it does not require octree. If what you are looking for is octree, AMReX could generate octree like grids, but does not store them in a tree type of data structure. Generally speaking, AMReX grids cannot be represented by a tree, because there is no direct paren/child relationship.

[BenWibking]

Yes, I was thinking primarily of the local adaptive timestepping in H-AMR, which does GRMHD and therefore doesn't have a Poisson solve. (Support for grid-level subcycling may also be planned for Enzo-E, but I don't think the code has been written yet.)

For my applications, I unfortunately do need a Poisson solve, and to be honest I haven't fully thought through how to do this correctly yet. I was imagining a local multigrid cycle on the subcycled grid, sacrificing strict consistency of the potential. Whether this would work in practice, I'm not sure.

Why would two (independently) subcycled grids at the same resolution level reduce at-scale performance? Shouldn't they be able to asynchronously communicate boundary data with each other while they are subcycling? I don't know how the boundary data exchange works in AMReX in detail, so perhaps I'm misunderstanding something here.

I am imagining something somewhat general, i.e. in principle any grid in the mesh could be independently subcycled.

[BenWibking]

@WeiqunZhang To clarify, the data structure does not matter to me. The ability to avoid multiple covering grids is important to me, and that is what enables local grid-level subcycling (as opposed to setting the timestep and subcycling on each refinement level). So as far as I understand, Berger-Oliger patch-based AMR can't support local grid-level subcycling due to the multiple covering grids, since the maximum signal speed anywhere on the mesh defines the root-level timestep.

[asalmgren]

What do you mean by multiple covering grids?

Sent from my iPhone

On Jul 1, 2020, at 4:04 PM, Ben Wibking notifications@github.com wrote:



@WeiqunZhang https://github.com/WeiqunZhang To clarify, the data structure does not matter to me. The ability to avoid multiple covering grids is important to me, and that is what enables local grid-level subcycling (as opposed to setting the timestep and subcycling on each refinement level). So as far as I understand, Berger-Oliger patch-based AMR can't support local grid-level subcycling due to the multiple covering grids, since the maximum signal speed anywhere on the mesh defines the root-level timestep.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/amrex/issues/1070#issuecomment-652687570, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YRCE3RQWRPLPL3BVCLRZO6IFANCNFSM4OL2ATXA .

[zingale]

do you mean that you want to ensure that the coarse grid under the hotspot is completely covered by the fine grid, so then you don't need to evolve the coarse grids at all? If this is so, then I think that means you don't want to subcycle through the level hierarchy, right?

[BenWibking]

I just mean multiple grids (of different resolution levels) sitting on top of the same part of the domain.

[BenWibking]

@zingale Yes, exactly. I don't want to evolve anything but the finest grid at any point in the domain. With traditional implementations of octree AMR, this is trivially avoided, since at any point in the domain, there is only one grid (i.e. there are no 'coarse grids').

[BenWibking]

@zingale Or at least I don't want to evolve anything but the finest grid at any point in specially-designated parts of the domain, such as where the code has identified hot spots.

[asalmgren]

It is definitely possible to do that in amrex - but are you willing to advance all the fine grids touching each other to advance with the same time step?

Sent from my iPhone

On Jul 1, 2020, at 4:31 PM, Ben Wibking notifications@github.com wrote:



@zingale https://github.com/zingale Or at least I don't want to evolve anything but the finest grid at any point in specially-designated parts of the domain, such as where the code has identified hot spots.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AMReX-Codes/amrex/issues/1070#issuecomment-652695030, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6YUEY4BMGE73UFA3RIDRZPBOFANCNFSM4OL2ATXA .

[BenWibking]

No, I want to avoid that if possible.

[BenWibking]

Is there any update on this?

[BenWibking]

@asalmgren @WeiqunZhang @zingale Just want to check again if this is a feature that will be feasible to implement with AMReX in the near-term. If this is not something that can be supported, I'll close this issue.