Fast Math Verification

[ax3l]

This issue documents a verification run for "fast math" (-ffast-math on gcc or --use_fast_math on nvcc).

I suspect that for long running simulations (e.g. 30k steps and more), it has a significant influence on final energy spectra.

Fast math is by default enabled in the PIConGPU GPU/cuda backend (via ALPAKA_CUDA_FAST_MATH) and by default disabled on the CPU backends (such as OpenMP/omp2b) where it needs to be passed by CXXFLAGS to control, e.g. via export CXXFLAGS="-g0 -O3 -m64 -ffast-math".

Method

Running the FoilLCT (a0=5, plane wave laser, 192 n_c, 1mu foil) example with 8.cfg repeatedly with varied:

• seed
• fast math (fm) options

runs: (first number as in cmakeFlags)

• 000: seed 42, +fm
• 001: seed 42, +fm
• 100: seed 29814, +fm
• 101: seed 29814, +fm
• 200: seed 74531, +fm
• 201: seed 74531, +fm
• 300: seed 42, -fm
• 301: seed 42, -fm
• 400: seed 29814, -fm
• 401: seed 29814, -fm
• 500: seed 74531, -fm
• 501: seed 74531, -fm

Diff to Default example

Commit

Branch: topic-fastMathFoilTest

Output

On HZDR file dirs in /bigdata/hplsim/development/huebl/foilLCT_fm/.

Version & Software

PIConGPU 0.4.2 on Hemera (HZDR) P100 with CUDA 9.2

[ax3l]

First test looks surprisingly stable after 60'000 steps (plane wave, a0=5):

Energy is not fully converged yet and one could do as follow-up tests:

• Gaussian laser pulse, non-periodic in x
• more a0
• even larger box in y
• select only forward or +/- 15 degree forward ions

[sbastrakov]

From your data the speedup due to fast math seems rather minor. Comparing configurations 000 and 300, calculation time is 38:52 vs. 39:32 and full simulation time is also about 40 seconds apart, in relative terms only about 1.7% speedup.

[ax3l]

Just for notes: I did a bit of I/O (text-based histograms every 100 steps) during the sim and one might want to compare pure simulation time without init. But I quite tend to agree that the "risk" of fast-math for the little currently seen speedup might not be worth it to make it "default on".

Nevertheless, the influence from the first view seems little and the next steps as outlined above need to be done to verify further.

Nevertheless, one has to verify that for several setups and without I/O. Further research welcome!