Closed sandroos closed 3 years ago
galfthan
commented
9 years ago What was the floating point precision here, DP SPF? If so the differences seem to be quite small. If you change the partitioning you will change the order of computation which will give rise to some differences.
In the case of 1 or 2 hyperthreads I guess you mean the number of processes identical and change the numbers of threads?
sandroos
commented
9 years ago vlsvdiff used here in now in branch https://github.com/fmihpc/vlasiator/tree/vlsvdiff_output
sandroos
commented
9 years ago DP SPF, yes.
And yes, these differences are still quite small, but this is literally after 10 time steps (0.1 seconds). I've usually checked the results after 20 seconds.
galfthan
commented
9 years ago How about DP DPF, did you test that?
sandroos
commented
9 years ago DP DPF:
sandroos
commented
9 years ago poisson vs master results are different with DP DPF, derp.
sandroos
commented
9 years ago On my laptop the difference between identical runs (but different # MPI processes) is
sandroos@sandroos12k:~/Codes/vlasiator$ ./vlsvdiff_DP fullf.0000010.vlsv ../d/fullf.0000010.vlsv avgs 0
INFO Reading in two files.
File names: fullf.0000010.vlsv & ../d/fullf.0000010.vlsv
NonIdenticalBlocks: 0
IdenticalBlocks: 704
Absolute_Error: 1.67023e-16
Mean-Absolute-Error: 2.6596e-19
Max-Absolute-Error: 1.38778e-17
Absolute-log-Error: 7.56468e-06
Mean-Absolute-log-Error: 1.20457e-08With the same #MPI processes I get identical results according to vlsvdiff. The max value of f is ~2e-10, so those differences are in the sixth decimal. I guess this is in-line with what we are seeing in Voima.
sandroos
commented
9 years ago Now that test_trans is mostly working, I ran some tests with master and varied the run parameters (#MPI and OMP threads). I used a single-core run (1 MPI, 1 OMP) as a reference solution.
The box is too small to tell how the differences really behave, but the differences are in the 7th decimal in rho (rho ~1e-4). Runs with one MPI process seem to produce identical results, irrespective of the number of OMP threads, so the caused by something related to MPI parallelization.
sandroos
commented
9 years ago This addition will reduce accuracy since it is done using floats (realf) instead of doubles (Real) right?
https://github.com/fmihpc/vlasiator/blob/master/vlasovsolver/cpu_trans_map.hpp#L712
(Didn't change test_trans results though)
sandroos
commented
9 years ago I'll ask this here:
what are the compiler flags used for the test package reference results? I don't seem to be able to reproduce the reference values with master.
I assume they are DP, DPF, VEC4D_AGNER, but what are the exact CXXFLAGS and MATHFLAGS ?
galfthan
commented
9 years ago DP, DPF, VEC4D_AGNE, and -O2 instead of -O3 and -ffast-math
I tested and got bitidentical results for master. This is the diff of makefiles:
diff --git a/MAKE/Makefile.voima b/MAKE/Makefile.voima
index 28293cc..c370f2c 100644
--- a/MAKE/Makefile.voima
+++ b/MAKE/Makefile.voima
@@ -20,10 +20,10 @@ FLAGS =
CC_BRAND = gcc
CC_BRAND_VERSION = 4.8.1
#CXXFLAGS += -O3 -fopenmp -funroll-loops -std=c++0x -W -Wall -Wno-unused -fabi-version=0 -mavx
-CXXFLAGS += -O3 -fopenmp -funroll-loops -std=c++0x -fabi-version=0 -mavx
+CXXFLAGS += -O2 -fopenmp -funroll-loops -std=c++0x -fabi-version=0 -mavx
-MATHFLAGS = -ffast-math
+MATHFLAGS =
LDFLAGS =
LIB_MPI = -lgomp
diff --git a/Makefile b/Makefile
index 9250242..fdeb1ea 100644
--- a/Makefile
+++ b/Makefile
@@ -6,10 +6,10 @@ include MAKE/Makefile.${ARCH}
FP_PRECISION = DP
#Set floating point precision for distribution function to SPF (single) or DPF (double)
-DISTRIBUTION_FP_PRECISION = SPF
+DISTRIBUTION_FP_PRECISION = DPF
#Set vector backend type for vlasov solvers, sets precision and length. Options: VEC4D_AGNER, VEC4F_AGNER, VEC8F_AGNER, VEC4D_FALLBACK, VEC4F_FALLBACK
-CXXFLAGS += -DVEC8F_AGNER
+CXXFLAGS += -DVEC4D_AGNER
#set a default archive utility, can also be set in Makefile.arch
AR ?= arI added a testpackage Makefile target, the flags are set in Makefile (and not in architecture-specific Makefile.arch).
ykempf
commented
3 years ago This was fixed then, and looked into quite a lot during our 2020 AMR debugging campaign.
So, I was testing if the poisson branch could be merged to master, i.e., if they would give identical results for a mini-magnetosphere (ABF) test. Of course they don't.
Then I started to test just by using the current master with different run parameters. The results are identical (vlsvdiff) between runs if
The results are different if
For example, here is vlsvdiff between RCB and BLOCK partitioning
And here is the difference in rho after just 10 timesteps at t=0.1 seconds. The differences are still small but they keep on growing.