Open ornellamarciano opened 3 years ago
That's a very thorough report, @ornellamarciano; thank you. Having scanned your code, what stands out to me is your use of tf.compat.v1.keras.layers.Embedding
. We have isolated a source of nondeterminism to the backprop into embedding matrices (in TensorFlow) and we're actively working on a patch to address this. We plan to release soon, at which point I'll ask you to test again with the patch.
Deeper information: the forward path of selecting word embeddings for a given batch is usually implemented with tf.gather
(and I've confirmed that this is true for tf.compat.v1.keras.layers.Embedding
). The backprop for tf.gather
produces a tf.IndexedSlices
gradient tensor which is reduced to a dense tensor, in order to update the embedding matrix, using tf.convert_to_tensor
. tf.convert_to_tensor
uses one of the segment sum ops, which we have confirmed operates nondeterministically. The upcoming patch, released in version 0.4.0 of the framework-determinism package and applied using fwd9m.tensorflow.enable_determinism()
, will dynamically patch the segment sum ops to make them operate deterministically for 16-bit floating-point and 32-bit floating-point operands. The patch will not address nondeterminism in 64-bit floating-point operands; that will be addressed by a later CUDA-level fix (I don't know when we'll get to that).
Thanks to my colleague @wenscarl for helping to isolate the source of nondeterminism and develop the patch for it.
@duncanriach @wenscarl Thanks a lot for your quick and detailed answer ! Regarding float64, I tried it to escape non-determinism due to a potential issue in floating point. But if with the new patch, I manage to get deterministic results, I will totally work with float32 (faster and less memory consumption). Thanks again.
Right, I would expect you to see a much smaller amount of noise accumulating when using float64, which seems to be what you witnessed.
Hi @duncanriach do you know when the patch solving the non determinism of embedding layers will be released ? Thanks for your help!
I don't have an ETA for you, but I can tell you that we're actively working on it and that's it's top priority. I'm hoping for a release in the next few weeks.
Hi @duncanriach do you have any update ? Thanks a lot!
Hi @ornellamarciano,
Thanks for checking-in.
From stock TensorFlow version 2.5 onwards, the use of TensorFlow's segment reduction ops running on a GPU, with the expectation of reproducibility (i.e. when TF_DETERMINISTIC_OPS is set to '1'
or 'true'
), will cause tf.errors.UnimplementedError
to be thrown, this includes back-propagating into a dense embedding matrix via the backwards path through tf.gather
(which uses tf.math.unsorted_segment_sum
). See stock TensorFlow pull request 47772 for more information.
@benbarsdell has developed deterministic GPU implementations of all the TensorFlow segment reduction ops, including tf.math.unsorted_segment_sum
. A pull request to add this to stock TensorFlow will be created soon (there are some dependencies that it's waiting for). I expect these changes to be released in stock TensorFlow version 2.7.
You might also want to try applying the unreleased patch from the cloned repo. See the discussion in issue 19.
Prerequisites
I am using latest TensorFlow NGC container nvcr.io/nvidia/tensorflow:20.09-tf2-py3
Issue
When running my Deep Learning model on 1 GPU (Tesla T4) within the NGC container 20.09, I don't get reproductible results between consecutive runs (different predictions) although all the seeds are set correctly. Running the same code on CPU gives reproductible results over consecutive runs.
On a small dataset (138k samples), I managed to get reproductible results within the NGC by setting the default float type to float64 (instead of float32). But with a bigger dataset (2.6m samples), after 1million of samples, the predictions start to differ over different runs. The more I increase the dataset size, the bigger is the difference between predictions of different consecutive runs.
My datasets are tfrecord files of hashed values, serialized by batch of 10,000 samples. Here are the small_dataset (size 13M): small_dataset and the big_dataset (size 251M) - can you send me your email to share it with you via Google Drive to be able to reproduce the issue? You need to unzip them before running the script bellow.
Command lines:
Script
Could you help me to understand why I don't get the exact same predictions after consecutive runs ?