GRTLCollaboration / GRChombo

An AMR based open-source code for numerical relativity simulations.
BSD 3-Clause "New" or "Revised" License
80 stars 49 forks source link

Add ParaView Catalyst in-situ visualization capability #222

Open mirenradia opened 1 year ago

mirenradia commented 1 year ago

This PR adds ParaView Catalyst in-situ visualization capabilities to GRChombo.

User documentation

The user documentation for this feature can be found in the README so, for brevity, I will not repeat that here. Instead I will focus on implementation details below. Once this PR is merged, I plan to add a version of this documentation page to the wiki.

Brief list and description of key changes

Add CatalystAdaptor class

This class translates the GRChombo grid into a vtkOverlappingAMR data structure that can be passed to ParaView. It also contains the functions to initialise ParaView/Catalyst and the pipelines, execute the user-provided pipelines at regular intervals and finalise ParaView/Catalyst at the end of the simulation.

Add Make.insitu file and cache.sh script

The Make.insitu makefile can be included in GRChombo example GNUMakefiles to add Catalyst capability to the example. It adds the necessary compiler and linker flags to do this using the cxxcppflags and XTRALDFLAGS Chombo makefile variables so these cannot be set in the Chombo Make.defs.local file or elsewhere. It gets these flags using the paraview-config Python script (which in turn calls cmake) which is provided by suitable ParaView builds/installations. Since this script takes a long time to run and the Chombo build system calls it several times, the cache.sh script caches the output to temporary files in /tmp.

Add changes to GRChombo in order to enable Catalyst support

Such changes include:

Note that all changes are conditional compiled on the USE_CATALYST macro which is defined by Make.insitu if the PARAVIEW_DIR environment variable is set so this shouldn't break building examples without Catalyst.

Add a Catalyst Insitu test

This test just creates a simple 1-level GRChombo grid with data set by the Polynomial class of the AMR Interpolator test. It then runs a simple Catalyst pipeline which outputs a slice of the variable A as a PNG file. This file is then compared with an expected image to verify there isn't a regression in the in-situ code. Note that if USE_CATALYST is undefined, the test is skipped (it does nothing and exits immediately).

Add a GitHub action to run the Catalyst Insitu test

Since building with Catalyst is a little more complicated, this test is run separately from all the others. It is built using a version of ParaView built by a GitHub action in the GRChombo/paraview-build repository.

Add Catalyst support to the BinaryBH example

There are a small number of modifications to the BinaryBH example in order to add Catalyst support. The main one is the inclusion of Make.insitu to the GNUMakefile, but there are also some changes to ensure Weyl4 is calculated in specificPostTimeStep() if Catalyst is activated and to call catalystCoProcess() at t=0.

Add some example Catalyst Python scripts

See the comment lines at the top of these scripts (in the Examples/BinaryBH/catalyst_scripts directory) to see what they do. They all take slices through the center (which is passed from GRChombo using environment variables) of constant z.

I appreciate this is a very large PR so please take your time and don't hesitate to ask me any questions you might have. I don't expect you to go through and understand all of the ParaView/Catalyst/VTK function calls/object instantiations but it would be good to check the GRChombo side of things and make sure things are consistent with the rest of the code.

amelialdrew commented 1 year ago

I was able to implement this successfully into one of my examples on Cambridge CSD3