volume rendering issues with AMR

[BenWibking]

I tried out the volume extract after you mentioned it in my other query. However, I'm getting some strange issues when I'm using it with an AMR mesh. I'm not sure whether this is due to the AMReX integration or something in Ascent itself.

This is a volume rendering of the temperature field, with a shock entering from the left boundary, hitting a uniform temperature box of gas, but with a refined region around a dense gas cloud. There's something wrong at the coarse-fine boundaries here:

(There's also a white line along the diagonal, but that goes away if I tilt the camera slightly away from its default location.)

The density field volume rendering looks like this:

[BenWibking]

Not sure if this is related, but I did notice this error message in the log file:

[Error] Ascent::publish 
file: /avatar/bwibking/ascent/src/ascent/runtimes/ascent_main_runtime.cpp
line: 2054
message: 
User specified Ghost field 'topo_ghosts' does not exist. Possible fields:  'gasDensity' 'x-GasMomentum' 'y-GasMomentum' 'z-GasMomentum' 'gasEnergy' 'scalar' 'radEnergy' 'x-RadFlux' 'y-RadFlux' 'z-RadFlux' 'temperature'

[cyrush]

IF the AMReX Multifab has ghosts elements (ngrow > 0)

https://github.com/AMReX-Codes/amrex/blob/bf79c456a9a5fe6b22f3ba51b5973cd44b03c244/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L431

Then the AMReX to Blueprint wrappers will create a ghost field that Ascent can use to remove duplicate data on domain boundries.

It's possible that this field is named ascent_ghosts instead of topo_ghosts, but I don't see that name the list above --so maybe ngrow == 0. It will also create nesting info that describes the AMR level relationships, so overlapping elements can be removed before rendering.

In past use, I suspect we have been testing with ngrow > 0 + nesting info, not ngrow=0 + nesting info. It could be a bug with how we compose the nesting before thresholding.

I suspect the white line is lost rays due to numerics, especially if a slight view change make it disappear.

Any chance you can share hdf5 extract files with us?

You could create them with yaml actions like this:

-
  action: "add_extracts"
  extracts:
    e1:
      type: "relay"
      params:
        path: "./conduit_blueprint"
        protocol: "blueprint/mesh/hdf5"

Some more info: (https://ascent.readthedocs.io/en/latest/Tutorial_Intro_Extracts.html#exporting-input-mesh-data-to-blueprint-hdf5-files)

We could run those though ascent's replay exe with the same yaml scene setup as you and try to understand the white line and the ghost issues.

[BenWibking]

I think you've found the issue. The way it's currently set up, when an Ascent render happens in the code, I create new MultiFabs on each level so that I can combine the state variables with derived variables in a single MultiFab with nGrow = 0 (https://github.com/BenWibking/quokka/blob/c671a5f088582cd5b2b64b559b9c65f7ecae0a2b/src/simulation.hpp#L1200), then pass these Multifabs to Conduit.

I built my copy of Conduit without HDF5, but I'll try rebuilding with Spack with HDF5 and save the Blueprint HDF5 files.

[BenWibking]

Here's the HDF5 extract files and ascent_actions.yaml: https://cloudstor.aarnet.edu.au/plus/s/gGkKDjdPCLi9EGK

[BenWibking]

I've also tried setting nGrow = 4, and it silences the warning messages about topo_ghosts, but the volume rendering is still wrong (see below).

The blueprint files are here: https://cloudstor.aarnet.edu.au/plus/s/QpSw1oeoSJTgSwm

[cyrush]

Thanks for the files.

I was able to take quick look, one thing I noticed is that the first file had topo_ghosts. Looking at this field in VisIt, it seems to be the distilled AMR info, which should help the issue b/c I still see it in your cycle 0 rendering.

The other extracts do not have the topo_ghosts field.

I checked and they do all have Conduit Blueprint AMR info (nestset), so it looks like the right AMR info is there.

Now that ngrow is > 0, I expected to see a field named ascent_ghosts but it is not in these files.

I need to look a bit more, but unfortunately it will be at least a week until I have some more time to invest.

[BenWibking]

This is still an issue with the current Ascent develop branch and the current VTK-m 2.1 ("release") branch. I've made a much smaller reproducer here: reproducer.tar.gz

I was hoping this might have been related to the VTK-m ghost zone/slicing issues, but it produces the same incorrect rendering as before:

There is also an error message printed when I run replay on this Blueprint file:

Root file conduit_blueprint.cycle_000000.root
[Error] Ascent::publish 
file: /Users/benwibking/ascent_build/conduit-v0.8.8/src/libs/conduit/conduit_node.cpp
line: 16226
message: 
Node::as_signed_long_long_array() -- DataType int32 at path domain_000000/fields/ascent_ghosts/values does not equal expected DataType int64