CPU/GPU results inconsistency of test problem Cosmic_Ray_Diffusion

[hsinhaoHHuang]

Description

The difference between the results of CPU and GPU is not in the machine precision for the test problem Cosmic_Ray_Diffusion.

Procedure to reproduce the issue

1. Modify the source code

1-1. cd src/ 1-2. vim TestProblem/Hydro/CR_Diffusion/Init_TestProb_Hydro_CR_Diffusion.cpp

• Comment out the error message in line 200-201 to run a Gaussian ball case,

  // if ( CR_Diffusion_Type == 0  ||  CR_Diffusion_Type == 1  ||  CR_Diffusion_Type == 2  ||  CR_Diffusion_Type == 3 )
  //   Aux_Error( ERROR_INFO, "CR_Diffusion_Type (%d) is not supported yet !! (Unless you have fixed the all the fluid except c...

  2. Compile

  2-1. cp ../example/test_problem/Hydro/CR_Diffusion/generate_make.sh . 2-2. vim generate_make.sh

• Add --openmp=false --mpi=false --bitwise_reproducibility=true --gpu=false --double=false for Case_CPU_SINGLE

2-3. sh generate_make.sh 2-4. make clean && make

3. Run

3-1. cd ../bin/ 3-2. cp -r ../example/test_problem/Hydro/CR_Diffusion . 3-3. cd CR_Diffusion 3-4. mv ../gamer . 3-5. vim Input__TestProb

• set CR_Diffusion_Type = 0 for a 3D Gaussian distribution
• set CR_Diffusion_Mag_Type = 0 for the uniform magnetic field
• set CR_Diffusion_V* = 4.0 for the background velocity

3-6. vim Input__Parameter

• set CR_DIFF_PARA = 0.00 to simplify problem
• set OPT__BC_FLU_* = 1 to support background velocity

3-7. ./gamer 1>>log 2>&1

4. Store the data

4-1. mkdir Case_SINGLE_CPU 4-2. mv Data_0000* Case_SINGLE_CPU/ 4-3. mv gamer Case_SINGLE_CPU/ 4-4. mv Record__* Case_SINGLE_CPU/ 4-5. cp Input__* Case_SINGLE_CPU/

5. Repeat for different combinations

5-1. Repeat Step 2 to Step 4 for the other three cases

• Case_SINGLE_GPU (--gpu=true --double=false)
• Case_DOUBLE_CPU(--gpu=false --double=true)
• Case_DOUBLE_GPU(--gpu=true --double=true)

6. Compare data

6-1. Compile the tool/analysis/gamer_compare_data for single precision and double precision and name them as GAMER_CompareData_SINGLE and GAMER_CompareData_DOUBLE 6-2. ./GAMER_CompareData_DOUBLE -m -c -i Case_DOUBLE_CPU/Data_000010 -j Case_DOUBLE_GPU/Data_000010 -o Compare_DOUBLE_CGPU 6-3. ./GAMER_CompareData_SINGLE -m -c -i Case_SINGLE_CPU/Data_000010 -j Case_SINGLE_GPU/Data_000010 -o Compare_SINGLE_CGPU

Results

Compare_SINGLE_CGPU:

#Level    PID1    PID2  (  i,  j,  k ) Field           Data1           Data2          AbsErr          RelErr
     0     343     343  (  0,  0,  0 )     0   1.4691863e+00   1.6292144e+00  -1.6002810e-01  -1.0329723e-01
     0     504     504  (  5,  5,  5 )     1   5.9106088e-01   7.5351155e-01  -1.6245067e-01  -2.4163915e-01
     0     504     504  (  5,  5,  5 )     2   5.9690022e-01   7.5477648e-01  -1.5787625e-01  -2.3360061e-01
     0     504     504  (  5,  4,  5 )     3   7.6659769e-02   4.7234055e-01  -3.9568079e-01  -1.4414592e+00
     0     343     343  (  2,  5,  5 )     4   6.4545189e+01   6.0991863e+01   3.5533257e+00   5.6609990e-02
     0     464     464  (  6,  6,  6 )     5   4.8183027e-01   1.0200931e+00  -5.3826281e-01  -7.1676469e-01
     0     336     336  (  2,  7,  6 )     6   4.5468292e+00   4.5850992e+00  -3.8269997e-02  -8.3815805e-03
     0     343     343  (  2,  5,  5 )     7   4.6186199e+00   4.6590672e+00  -4.0447235e-02  -8.7192497e-03
     0     336     336  (  7,  6,  2 )     8   4.5467596e+00   4.5850220e+00  -3.8262367e-02  -8.3800443e-03
     0     504     504  (  6,  5,  6 )  1000   4.3393788e+00   4.4320211e+00  -9.2642307e-02  -2.1123722e-02
     0     504     504  (  5,  6,  6 )  1001   4.3421907e+00   4.4328880e+00  -9.0697289e-02  -2.0671561e-02
     0     504     504  (  5,  5,  6 )  1002   4.1058764e+00   4.2488337e+00  -1.4295721e-01  -3.4221944e-02

Compare_DOUBLE_CGPU:

#Level    PID1    PID2  (  i,  j,  k ) Field           Data1           Data2          AbsErr          RelErr
     0     504     504  (  5,  6,  6 )     0   3.3092350e-01   3.8743710e-01  -5.6513593e-02  -1.5734046e-01
     0     504     504  (  2,  6,  6 )     1   7.6786533e-01  -6.0994049e-01   1.3778058e+00   1.7448881e+01
     0     504     504  (  6,  2,  6 )     2  -2.3344095e-01   8.4463682e-01  -1.0780778e+00  -3.5277653e+00
     0     504     504  (  6,  7,  3 )     3  -4.3214597e-01   4.0828531e-01  -8.4043128e-01   7.0444943e+01
     0     504     504  (  6,  3,  5 )     4   6.2521044e+01   4.3762306e+01   1.8758738e+01   3.5299485e-01
     0     504     504  (  3,  5,  6 )     5   1.6097461e+01   2.0103033e+01  -4.0055729e+00  -2.2129935e-01
     0     504     504  (  4,  5,  6 )     6   4.6092457e+00   5.5613991e+00  -9.5215340e-01  -1.8723560e-01
     0     504     504  (  6,  0,  6 )     7   4.8670427e+00   4.1642441e+00   7.0279859e-01   1.5563642e-01
     0     504     504  (  6,  6,  4 )     8   5.3790985e+00   4.5633209e+00   8.1577757e-01   1.6410041e-01
     0     504     504  (  5,  5,  6 )  1000   4.3230341e+00   5.6671486e+00  -1.3441145e+00  -2.6908707e-01
     0     504     504  (  6,  1,  6 )  1001   4.9551557e+00   3.9974120e+00   9.5774367e-01   2.1395955e-01
     0     504     504  (  6,  5,  5 )  1002   5.6527335e+00   4.4445403e+00   1.2081932e+00   2.3931077e-01

Summary of tests I have done.

1

• The difference is small initially, and the problem will appear suddenly between a few steps.

• It is reproducible by rerunning it or restarting it from the middle snapshot.

• The difference still exists when there is no OpenMP, no MPI, FLU_BLOCK_SIZE_X=1, FLU_GPU_NPGROUP=1, and no AMR.

2

• The difference still exists even with CR_DIFF_*= 0.0.

• The difference still exists even with compilation option CR_DIFFUSION turned off.

• The problem seems related to the magnetic field, especially when the magnetic field is too strong.

  • However, the MHD test problems BlastWave with MHD andMHD_OrszagTangVortex do not have this issue, as far as I have tested.

3

• The difference still exists even with a non-zero background velocity and a three-dimensional magnetic field.

  • In some cases, adding a background velocity can help. The difference with double precision is 5 orders of magnitude smaller than the difference with single precision. But not all the cases can be solved by this way.

• src/Model_Hydro/CPU_Hydro/CPU_Shared_ConstrainedTransport.cpp:411 , in dE_Upwind() if ( FABS(FC_Vel) <= MAX_ERROR )-> if ( true ) : For some cases without background velocities, it can reduce the difference and the difference with double precision is 7 orders of magnitude smaller than the difference with single precision. But in general, it doesn't solve all the cases.

4

• Turning on GAMER_DEBUG does not print anything weird.

More details about test problem related parameters

• For CR_Diffusion_Type=0 (Gaussian distribution ball) and CR_Diffusion_Mag_Type=0(uniform magnetic field),

  • The difference is in machine precision for standard 2D diffusion, CR_Diffusion_G{X,Y,Z}={1,1,0} and CR_Diffusion_Mag{X,Y,Z}={5,5,0}

    • The problem appears when the Gaussian is 3D, CR_Diffusion_G{X,Y,Z}={1,1,1} and CR_Diffusion_Mag{X,Y,Z}={5,5,0}

    • The problem appears when the magnitude of the magnetic field becomes large, CR_Diffusion_G{X,Y,Z}={1,1,0} and CR_Diffusion_Mag{X,Y,Z}={10,10,0}

  • The difference is in machine precision for CR_Diffusion_G{X,Y,Z}={1,1,0} and CR_Diffusion_Mag{X,Y,Z}={1,0,0}

    • The problem appears when the magnitude of the magnetic field becomes large, CR_Diffusion_G{X,Y,Z}={1,1,0} and CR_Diffusion_Mag{X,Y,Z}={5,0,0}

[hsinhaoHHuang]

Although I am not sure whether this is the same issue, I found the results with GAMER_DEBUG ON are not identical with GAMER_DEBUG OFF when using GPU (BITWISE_REPRODUCIBILITY is on), while they can be identical when using CPU. This also happens in the test problem MHD_OrszagTangVortex.

[hyschive]

@JeiwHuang It is expected that GAMER_DEBUG may change the simulation results on the machine-precision level since it enables several additional checks in the fluid solvers.