Open mtaillefumier opened 5 months ago
So far for the AlCl3 system (TF GPU DeepMD CPU only)
TF_DETERMINISM_OPS=1, OMP_NUM_THREADS=1, TF_INTER_OP_PARALLELISM_THREADS=1, TF_INTRA_OP_PARALLELISM_THREADS=1
: I run the training three times from scratch (100k straining steps) and the difference of atomic forces between the three runs compared pairwise is ZERO (10e-18).We can run DeepMD cpu deterministically
TF_DETERMINISM_OPS=1, OMP_NUM_THREADS=2, TF_INTER_OP_PARALLELISM_THREADS=1, TF_INTRA_OP_PARALLELISM_THREADS=1
Use the variable TF_DETERMINISM_OPS to turn on deterministic paths in tensorflow. They are three other variables impacting the parallelism
OMP_NUM_THREADS
,TF_INTER_OP_PARALLELISM_THREADS
, andTF_INTRA_OP_PARALLELISM_THREADS
that also need to be checked.So far we want to check with this configuration, TF on GPU, deepmd cpu only (we can make thing more complicated afterwards with a full GPU stack)
Again we want to check the variability in the atomic forces between the different runs not the comparison between predicted and DFT data. This can be postponed for later.