Turn on TF_DETERMINISM_OPS

[mtaillefumier]

Use the variable TF_DETERMINISM_OPS to turn on deterministic paths in tensorflow. They are three other variables impacting the parallelism OMP_NUM_THREADS, TF_INTER_OP_PARALLELISM_THREADS, and TF_INTRA_OP_PARALLELISM_THREADS that also need to be checked.

So far we want to check with this configuration, TF on GPU, deepmd cpu only (we can make thing more complicated afterwards with a full GPU stack)

• AlCl3
• NUC
• https://zenodo.org/records/7234728 (water results)
• Boroosilicate data (https://zenodo.org/records/7577034). This one will be the object of a separate task.

Again we want to check the variability in the atomic forces between the different runs not the comparison between predicted and DFT data. This can be postponed for later.

[mtaillefumier]

So far for the AlCl3 system (TF GPU DeepMD CPU only)

• TF_DETERMINISM_OPS=1, OMP_NUM_THREADS=1, TF_INTER_OP_PARALLELISM_THREADS=1, TF_INTRA_OP_PARALLELISM_THREADS=1: I run the training three times from scratch (100k straining steps) and the difference of atomic forces between the three runs compared pairwise is ZERO (10e-18).

We can run DeepMD cpu deterministically

• TF_DETERMINISM_OPS=1, OMP_NUM_THREADS=2, TF_INTER_OP_PARALLELISM_THREADS=1, TF_INTRA_OP_PARALLELISM_THREADS=1