PrincetonUniversity / athena-public-version

(MOVED) Athena++ GRMHD code and adaptive mesh refinement (AMR) framework. Active repository --->
https://github.com/PrincetonUniversity/athena
BSD 3-Clause "New" or "Revised" License
160 stars 117 forks source link

Help understanding energy conservation with FFT self Gravity #50

Closed umgraftw closed 4 years ago

umgraftw commented 4 years ago

Greetings,

According to user @pdmullen in #15 one should not expect energy to be conserved to machine precision using the in-built FFT gravity solver, and pdmullen further provided a graph showing a comparison of E{tot,0} to E{tot} versus time.

I am noticing that my simulations are more extreme with their change in total energy versus time, which is concerning for me.

I was hoping that someone could help me to understand why energy isn't conserved in the FFT self gravity solver and under what circumstances that should be concerning to me (ie can I trust the dynamics of the system?)

Here is a simple test we did:

// C++ headers
#include <cmath>      // sqrt()
#include <iostream>   // endl
#include <sstream>    // stringstream
#include <stdexcept>  // runtime_error
#include <string>     // c_str()
#include <time.h>     // srand(time)

// Athena++ headers
#include "../athena.hpp"
#include "../athena_arrays.hpp"
#include "../bvals/bvals.hpp"
#include "../coordinates/coordinates.hpp"
#include "../eos/eos.hpp"
#include "../field/field.hpp"
#include "../hydro/hydro.hpp"
#include "../mesh/mesh.hpp"
#include "../parameter_input.hpp"
#include "../gravity/gravity.hpp"

namespace {
Real gmma1, dl, pl, ul, BigMass;
Real bxl, byl, bzl;
Real gconst;
} // namespace

//========================================================================================
//! \fn void MeshBlock::ProblemGenerator(ParameterInput *pin)
//  \brief Problem Generator for the shock-cloud interaction test
//========================================================================================

void MeshBlock::ProblemGenerator(ParameterInput *pin) {
  Real gmma  = peos->GetGamma();
  gmma1 = gmma - 1.0;

    Real xmin = pin->GetReal("mesh","x1min");
    Real ymin = pin->GetReal("mesh","x2min");
    Real zmin = pin->GetReal("mesh","x3min");
    Real xmax = pin->GetReal("mesh","x1max");
    Real ymax = pin->GetReal("mesh","x2max");
    Real zmax = pin->GetReal("mesh","x3max");

    Real Lx = xmax - xmin;
    Real Ly = ymax - ymin;
    Real Lz = zmax - zmin;

  // Read input parameters
  Real rad    = pin->GetReal("problem","radius");
  Real pow    = pin->GetReal("problem","turbPower");
  Real rhoCen    = pin->GetReal("problem","rhoCen");
  Real mach = pin->GetReal("problem","Mach");
  Real iso_sp = pin->GetReal("hydro","iso_sound_speed");

  // Set paramters in ambient medium 
  Real drat = pin->GetReal("problem","drat");
  Real dr = pin->GetReal("problem","dr");
    Real ux = 0.0;
    Real uy = 0.0;
    Real uz = 0.0;
    Real vx = 0.0;
    Real vy = 0.0;
    Real vz = 0.0;
    Real x = 0.0;
    Real y = 0.0;
    Real z = 0.0;
    Real rho = 0.0;
    Real prCyl = pin->GetReal("problem","pres");
    Real prat = pin->GetReal("problem","prat");
    Real pr = 0.0;

  // Initialize the grid
  for (int k=ks; k<=ke; k++) {
    for (int j=js; j<=je; j++) {
      for (int i=is; i<=ie; i++) {
          x = pcoord->x1v(i);
          y = pcoord->x2v(j);
          z = pcoord->x3v(k);
          vx = pow*cos(2*PI*(x+Lx/2)/Lx)*sin(2*PI*abs(y + Ly/2)/Ly);
          vy = pow*(-1.0*sin(2*PI*(x+Lx/2)/Lx)*cos(2*PI*abs(y + Ly/2)/Ly));
        // cloud interior
        Real diag = std::sqrt(SQR(pcoord->x1v(i)) + SQR(pcoord->x2v(j)));
    Real profile =std::pow((1+SQR(diag)/8),-2);
        if (diag < rad) {
            rho = rhoCen;
            pr = prCyl;
        }
        else
        {
            rho = dr;
            pr = prat*prCyl;
        }
          ux = (1.0 - (rho - dr)/rhoCen)*vx;
          uy = (1.0 - (rho - dr)/rhoCen)*vy;
          uz = (1.0 - (rho - dr)/rhoCen)*vz;
        phydro->u(IDN,k,j,i) = rho;
          phydro->u(IM1,k,j,i) = rho*ux;
        phydro->u(IM2,k,j,i) = rho*uy;
        phydro->u(IM3,k,j ,i) = rho*uz;
        phydro->u(IEN,k,j,i) = pr/gmma1 + 0.5*rho*ux*ux + 0.5*rho*uy*uy + 0.5*rho*uz*uz;
      }
    }
  }

  return;
}

//========================================================================================
//! \fn void Mesh::InitUserMeshData(ParameterInput *pin)
//  \brief Function to initialize problem-specific data in mesh class.  Can also be used
//  to initialize variables which are global to (and therefore can be passed to) other
//  functions in this file.  Called in Mesh constructor.
//========================================================================================

void Mesh::InitUserMeshData(ParameterInput *pin) {
  if (SELF_GRAVITY_ENABLED) {
    //Self Gravity settings
    //Set 4piG
    Real fourPiG = pin->GetReal("problem","fourPiG");
    SetFourPiG(fourPiG);

    //Get total mass to set mean density
    Real rad = pin->GetReal("problem","radius");
    Real Lx = pin->GetReal("problem","Lx");
    Real Ly = pin->GetReal("problem","Ly");
    Real Lz = pin->GetReal("problem","Lz");
    Real xmin = pin->GetReal("mesh","x1min");
    Real ymin = pin->GetReal("mesh","x2min");
    Real zmin = pin->GetReal("mesh","x3min");  
    Real ie = pin->GetReal("mesh","nx1");
    Real je = pin->GetReal("mesh","nx2");
    Real ke = pin->GetReal("mesh","nx3");
    Real drat = pin->GetReal("problem","drat");
    Real rhoCen = pin->GetReal("problem","rhoCen");
    Real amb = pin->GetReal("problem","dr");
    for(int j=0;j<=je-1;++j){
      for(int i=0;i<=ie-1;++i){
        Real diag_sq = SQR((Lx/ie)*(i+0.5)+xmin) + SQR((Ly/je)*(j+0.5)+ymin);
        Real profile = rhoCen*std::pow((1+diag_sq/8),-2);
        if(std::sqrt(diag_sq)<rad){
          BigMass+=rhoCen*(Lx/ie)*(Ly/je)*(Lz/ke);
        } else {
          BigMass+=amb*(Lx/ie)*(Ly/je)*(Lz/ke);
        }
      }
    }

    //Real eps = pin->GetOrAddReal("problem","grav_eps", 0.0);
    //SetGravityThreshold(eps);
    SetMeanDensity(BigMass/(Lx*Ly*Lz));
  }
  return;
}

//----------------------------------------------------------------------------------------

And here is the input file

<job>
problem_id = Cylinder      # problem ID: basename of output filenames

<output1>
file_type  = hst        # History data dump
dt         = 0.1       # time increment between outputs

<output2>
file_type  = vtk        # data dump
variable   = prim       # variables to be output
dt         = 1.0        # time increment between outputs

<time>
cfl_number = 0.4        # The Courant, Friedrichs, & Lewy (CFL) Number
nlim       = 500000     # cycle limit
tlim       = 80.0      # time limit
integrator  = vl2       # time integration algorithm
xorder      = 2         # order of spatial reconstruction
ncycle_out  = 1         # interval for stdout summary info

<mesh>
nx1        = 400        # Number of zones in X1-direction
x1min      = -100.0       # minimum value of X1
x1max      = 100.0        # maximum value of X1
ix1_bc     = periodic     # inner-X1 boundary flag
ox1_bc     = periodic     # outer-X1 boundary flag

nx2        = 400        # Number of zones in X2-direction
x2min      = -100.0       # minimum value of X2
x2max      = 100.0        # maximum value of X2
ix2_bc     = periodic     # inner-X2 boundary flag
ox2_bc     = periodic     # outer-X2 boundary flag

nx3        = 1           # Number of zones in X3-direction
x3min      = -0.5        # minimum value of X3
x3max      = 0.5         # maximum value of X3
ix3_bc     = periodic    # inner-X3 boundary flag 
ox3_bc     = periodic    # outer-X3 boundary flag

<meshblock>
nx1 = 100
nx2 = 50
nx3 = 1

<hydro>
iso_sound_speed = 1.0 
gamma      = 1.66667    # gamma = C_p/C_v

<problem>
Mach       = 0.0       # Mach number of shock
rhoCen = 1.0
drat       = 0.1   # density ratio of cloud
prat = 1.0
beta       = 1.0       # plasma beta in upstream gas
turbPower = 0.0 
fourPiG    = 1.0       #four pi g
radius     = 50.0      #cloud radius
pres = 1.0
dr = 0.1
Lx         = 400
Ly         = 400
Lz         = 1
xblocks    = 1000
yblocks    = 1000
zblocks    = 1000

We are hoping to study the dynamics of a cylindrical filament in a turbulent rarefied medium. Any help understanding the change in total energy with the FFT self gravity solver would be greatly appreciated.

Thanks!

pdmullen commented 4 years ago

Greetings, @umgraftw!

I hope you are staying safe/healthy in these scary times... I have been thinking about energy conservation with self-gravity recently too, so maybe I can help tackle this question!

(1) A small update: in the Athena++ public repository, when history outputs are enabled, the gravitational potential energy is computed when self-gravity is enabled here: https://github.com/PrincetonUniversity/athena-public-version/blob/d535095157c9b6dd67829dab4ce605d21da6efbb/src/outputs/history.cpp#L101-L105 However, this is using the incorrect gravitational potential. The potential used in the above is the potential associated with the density distribution before executing the final stage of TimeIntegratorTaskList. The potential we truly desire here is the potential associated with the density distribution after executing the final stage. Therefore, in the development repository for Athena++ we have moved the potential solve to after executing ptlist->DoTaskListOneStage(). You can apply this hotfix by changing line 449-455 of main.cpp to

    for (int stage=1; stage<=ptlist->nstages; ++stage) {
      ptlist->DoTaskListOneStage(pmesh, stage);
      if (SELF_GRAVITY_ENABLED == 1) // fft (flag 0 for discrete kernel, 1 for continuous)
        pmesh->pfgrd->Solve(stage, 0);
      else if (SELF_GRAVITY_ENABLED == 2) // multigrid
        pmesh->pmgrd->Solve(stage);
    }

This fix will likely be implemented in the next public release. If you have been analyzing energy conservation with history outputs, this fix may produce some change (although I don't think this is the major issue here).

(2) I didn't check your pgen and athinput combination very closely (apologies), but one thing I would urge you to recheck and be absolutely confident of is that the mean density that you set with SetMeanDensity() is the mean density of the entire mesh. The mean density appears in the divergence of the gravitational stress tensor when applying Jeans' swindle. If you don't set this correctly, you will see conservation issues (see Issue #36).

(3) Even after you fix (1) and potentially address (2), total energy will not be conserved to machine precision in the public version of Athena++ for self-gravitating flows. To clear a potential point of confusion, this has nothing to do with the FFT Poisson solver, rather, the issue arises from how momenta and total energy updates from self-gravity are handled. Jiang et al. (2013) (https://ui.adsabs.harvard.edu/abs/2013NewA...19...48J/abstract) described a numerical scheme that can guarantee momentum and total energy conservation (to round-off error) by taking the divergence of momentum and energy "gravity fluxes". Athena++ updates the momentum (due to gravity) using such fluxes, but has not implemented the energy analogue (the energy gravity flux is a bit more involved, see Jiang et al. 2013 for details). Hanawa (2019) (https://ui.adsabs.harvard.edu/abs/2019JPhCS1225a2015H/abstract) describes a numerical scheme where one can actually guarantee momentum and total energy conservation if you apply gravity source terms in a special way. Again, this is not the scheme employed by the public version of Athena++.

(4) It is difficult to address when departures from total energy conservation "should be concerning". This is highly problem specific. Again, I would point you to Jiang et al. 2013 to see some of the consequences of violating total energy conservation.

(5) I have recently forked the private repo of Athena++ and developed a branch that applies the Hanawa (2019) numerical scheme for self-gravitating flows. The results are promising. For the linear wave Jean's instability problem (with lambda = 2 lambda_J), the Hanawa (2019) scheme conserves total energy to ~round-off error. This particular problem setup follows the collapse of an unstable wave into a sheet, and subsequently into a cylinder. The plots below show the different components of the energy during the collapse and the total energy conservation, respectively: energy_breakdown energy_conservation The same plot with the public version of Athena++ has total energy varying dramatically. These development efforts are still in early stages, but I am happy to chat about them if you are interested.

(6) Finally, this may be a bit off-the-wall (and may seem counterintuitive), but maybe you could try turning off gravity momentum fluxes in your Athena++ clone (comment out AddGravityFlux() in calculate_fluxes.cpp), and implement a momentum source term for self-gravity (i.e., in src/hydro/srcterms/self_gravity.cpp). For example, for X1-momentum add

cons(IM1,k,j,i) -= dtodx1*prim(IDN,k,j,i)*(phir-phil);

How does this change affect your energy conservation? (If you perform this test, I recommend that you use the rk2 integrator). Hanawa (2019) find that the gravity flux treatment can produce large errors in low density regions subject to strong gravity. This effect does not plague the source term treatment. Maybe this somehow connects to your energy conservation troubles (i.e., maybe these errors lead to hitting a density/pressure floor frequently?).

Yikes! Sorry for the water hose....hopefully something here helps.

Patrick

umgraftw commented 4 years ago

Greetings Patrick,

Thank you so much for all your advice and insight. I'm still processing what you've said but I didn't want to leave it any longer before thanking you.

As I said in my previous issue #36 I'm still quite new to using simulations, any and all help is greatly appreciated.

Thank you very much,

Will