GRMHD executable fails at high(ish) resolution

[nilsvu]

Bug reports:

The error All primitive inversion schemes failed at s = 0 sometimes occurs when h- or p-refining. For instance, the following run fails consistently and at resolutions small enough to run on a laptop:

• Executable: EvolveValenciaDivClean
• AnalyticSolution: KomissarovShock
• Limiter: SimpleWENO

Input File:

InitialTime: 0.0
InitialTimeStep: 0.01
FinalTime: 2.0

DomainCreator:
  Brick:
    LowerBound: [-0.5, 0.0, 0.0]
    UpperBound: [2.0, 1.0, 1.0]
    IsPeriodicIn: [false, true, true]
    InitialRefinement: [6, 0, 0]
    InitialGridPoints: [4, 2, 2]

AnalyticSolution:
  AdiabaticIndex: 1.3333333333333333
  LeftDensity: 1.
  RightDensity: 3.323
  LeftPressure: 10.
  RightPressure: 55.36
  LeftVelocity: [0.8370659816473115, 0., 0.] # u^i = [1.53, 0., 0.]
  RightVelocity: [0.6202085442748952, -0.44207111995019704, 0.] # u^i = [0.9571, -0.6822, 0.]
  LeftMagneticField: [10., 18.28, 0.]
  RightMagneticField: [10., 14.49, 0.]
  ShockSpeed: 0.5

TimeStepper: RungeKutta3

VolumeFileName: "./runs_komissarov_shock/h6_p4_weno/Volume"
ReductionFileName: "./runs_komissarov_shock/h6_p4_weno/Reductions"

NumericalFluxParams:

DampingParameter: 0.0

SlopeLimiterParams:

FixConservatives:
  CutoffD: 1.0e-12
  MinimumValueOfD: 1.0e-12
  SafetyFactorForB: 1.0e-12
  SafetyFactorForS: 1.0e-12

FixToAtmosphere:
  DensityOfAtmosphere: 1.0e-12
  DensityCutoff: 1.0e-12

ObserveNSlabs: 10
ObserveAtT0: true

Output:

Charm++: standalone mode (not using charmrun)
Charm++> Running in Multicore mode:  4 threads
Converse/Charm++ Commit ID: v6.8.0-0-ga36028edb
Warning> Randomization of virtual memory (ASLR) is turned on in the kernel, thread migration may not work! Run 'echo 0 > /proc/sys/kernel/randomize_va_space' as root to disable it, or try running with '+isomalloc_sync'.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 unique compute nodes (4-way SMP).
Charm++> cpu topology info is gathered in 0.066 seconds.

Executing './bin/EvolveValenciaDivClean' using 4 processors.
Date and time at startup: Wed Jan  2 16:03:25 2019

SpECTRE Build Information:
Version:                      0.0.0
Compiled on host:             f46c89f7904e
Compiled in directory:        /Users/nlf/Projects/spectre/spectre/build
Source directory is:          /Users/nlf/Projects/spectre/spectre
Compiled on git branch:       evolve_komissarov
Compiled with git hash:       f2f2ca3b6ebd525310a920f9a563bd644c2cad1c
Linked on:                    Wed Jan  2 13:09:04 2019

############ ERROR ############
Node: 0 Proc: 0
Line: 159 of /Users/nlf/Projects/spectre/spectre/src/Evolution/Systems/GrMhd/ValenciaDivClean/PrimitiveFromConservative.cpp
Function: static void grmhd::ValenciaDivClean::PrimitiveFromConservative<brigand::list<grmhd::ValenciaDivClean::PrimitiveRecoverySchemes::NewmanHamlin, grmhd::ValenciaDivClean::PrimitiveRecoverySchemes::PalenzuelaEtAl>, 2>::apply(gsl::not_null<Scalar<DataVector> *>, gsl::not_null<Scalar<DataVector> *>, gsl::not_null<tnsr::I<DataVector, 3, Frame::Inertial> *>, gsl::not_null<tnsr::I<DataVector, 3, Frame::Inertial> *>, gsl::not_null<Scalar<DataVector> *>, gsl::not_null<Scalar<DataVector> *>, gsl::not_null<Scalar<DataVector> *>, gsl::not_null<Scalar<DataVector> *>, const Scalar<DataVector> &, const Scalar<DataVector> &, const tnsr::i<DataVector, 3, Frame::Inertial> &, const tnsr::I<DataVector, 3, Frame::Inertial> &, const Scalar<DataVector> &, const tnsr::ii<DataVector, 3, Frame::Inertial> &, const tnsr::II<DataVector, 3, Frame::Inertial> &, const Scalar<DataVector> &, const EquationsOfState::EquationOfState<true, ThermodynamicDim> &) [OrderedListOfPrimitiveRecoverySchemes = brigand::list<grmhd::ValenciaDivClean::PrimitiveRecoverySchemes::NewmanHamlin, grmhd::ValenciaDivClean::PrimitiveRecoverySchemes::PalenzuelaEtAl>, ThermodynamicDim = 2]
All primitive inversion schemes failed at s = 0.
total_energy_density = 205.8120027478707
momentum_density_squared = (1.070095959268972e-18,33136.20054739921,22136.69421010788,2598369.417233342,1.070095959268972e-18,33136.20054739921,22136.69421010788,2598369.417233342,1.070095959268972e-18,33136.20054739921,22136.69421010788,2598369.417233342,1.070095959268972e-18,33136.20054739921,22136.69421010788,2598369.417233342)
momentum_density_dot_magnetic_field = (4.196504320146092e-09,1679.362672470281,-2421.174531306511,4160.924631288078,4.196504320146092e-09,1679.362672470281,-2421.174531306511,4160.924631288078,4.196504320146092e-09,1679.362672470281,-2421.174531306511,4160.924631288078,4.196504320146092e-09,1679.362672470281,-2421.174531306511,4160.924631288078)
magnetic_field_squared = (411.6240054953278,559.4031135696932,1070.390159677657,7339.277551896044,411.6240054953278,559.4031135696932,1070.390159677657,7339.277551896044,411.6240054953278,559.4031135696932,1070.390159677657,7339.277551896044,411.6240054953278,559.4031135696932,1070.390159677657,7339.277551896044)
rest_mass_density_times_lorentz_factor = 1e-12
previous_rest_mass_density = 5.685410615054082
previous_pressure = 214.421801713336
previous_lorentz_factor = 1.089862966378939

############ ERROR ############

Environment:

CMake Version: 3.9.1
SpECTRE Version: 0.0.0
SpECTRE branch: evolve_komissarov
SpECTRE hash: f2f2ca3b6ebd525310a920f9a563bd644c2cad1c
Hostname: f46c89f7904e
Host system: Linux-4.9.93-linuxkit-aufs
Host system version: 4.9.93-linuxkit-aufs
Host system processor: x86_64
CMAKE_CXX_FLAGS:  -fPIC
CMAKE_CXX_LINK_FLAGS: 
CMAKE_CXX_FLAGS_DEBUG: -g
CMAKE_CXX_FLAGS_RELEASE: -O3 -DNDEBUG
CMAKE_C_FLAGS:  -fPIC
CMAKE_C_LINK_FLAGS: 
CMAKE_C_FLAGS_DEBUG: -g
CMAKE_C_FLAGS_RELEASE: -O3 -DNDEBUG
CMAKE_Fortran_FLAGS: 
CMAKE_Fortran_LINK_FLAGS: 
CMAKE_Fortran_FLAGS_DEBUG: -g
CMAKE_Fortran_FLAGS_RELEASE: -O3 -DNDEBUG -O3
CMAKE_EXE_LINKER_FLAGS: 
CMAKE_BUILD_TYPE: Debug
CMAKE_CXX_COMPILER: /usr/local/bin/clang++
CMAKE_CXX_COMPILER_VERSION: 5.0.0
CMAKE_C_COMPILER: /usr/local/bin/clang
CMAKE_C_COMPILER_VERSION: 5.0.0
CMAKE_Fortran_COMPILER: /usr/bin/gfortran
CMAKE_Fortran_COMPILER_VERSION: 7.2.0
Python Version: 2.7.14
Charm Version:  6.8.0
Charm SMP:  YES
BLAS_LIBRARIES:  /usr/lib/x86_64-linux-gnu/libopenblas.so
Blaze Version:  3.2
Boost Version:  1.62.0
Catch Version:  2.1.0
Google Benchmark Found
GSL Version:  2.5
HDF5 Version:  1.10.0.1
jemalloc Version:  ..
LIBXSMM Version:  ..
LAPACK_LIBRARIES:  /usr/lib/x86_64-linux-gnu/libopenblas.so;/usr/lib/x86_64-linux-gnu/libopenblas.so
NumPy Version:  1.15.4
SciPy Version:  1.1.0

[nilsvu]

Call with @fmahebert and @nilsdeppe:

Behaviour of SimpleWENO with nonlinear elements could be relevant here:

[kidder]

Inversion test failure (too tight tolerance) /work/spectre/tests/Unit/TestingFramework.hpp:137: FAILED: CHECK( a == appx(b) ) with expansion: 0.0010928576 == Approx( 0.0010928576 ) with messages: seed = 1078102955 /work/spectre/tests/Unit/Evolution/Systems/RelativisticEuler/Valencia/ Test_PrimitiveFromConservative.cpp:200: expected_specific_internal_energy == specific_internal_energy a: T()=1.092857623199727313e-03 b: T()=1.092857623568477266e-03 a: 1.092857623199727313e-03 b: 1.092857623568477266e-03

[nilsdeppe]

Is there an action item here @kidder ? Can we close this since we've done a complete rewrite on how we do GRMHD?

[nilsdeppe]

Closing since those limiters aren't very robust to begin with and p=3 is not something they can handle.