Open yanniskar opened 2 years ago
Hi @yanniskar, thank you for isolating this issue, and for providing a reproducer. Please will you open an issue against the stock TensorFlow repo for this. Please reference this current issue. Once you have opened that, I will close this issue.
Also @yanniskar, for my records of customers who value this work, please can I know a bit more about you? Please could you give me your name and/or your affiliation?
Hi @yanniskar, thank you for isolating this issue, and for providing a reproducer. Please will you open an issue against the stock TensorFlow repo for this. Please reference this current issue. Once you have opened that, I will close this issue.
Done: https://github.com/tensorflow/tensorflow/issues/53846. To answer your other question, here is my linkedin: https://www.linkedin.com/in/yannis-karakozis-746488116/
Thanks for the help on this one :)
Thank you, and my pleasure.
BTW, from TensorFlow version 2.7 onwards, you no longer need to manually serialize (num_parallel_calls=1
) your dataset.map
preprocessing. @reedwm added a step that ensures that that functionality is deterministic and, in the future, as parallelized as possible (by splitting the stateless, computationally-intense functionality away from the stateful, computationally-mild functionality and parallelizing the former).
Problem Overview
I train my model on the same dataset in two different setups: A) single-gpu, B) multi-gpu. The former leads to deterministic results, the latter leads to non-deterministic results. The moment I replace the
tf.data.experimental.sample_from_datasets
API call with a direct call totf.data.Datasets
, B also becomes determinsitic.Environment
Python: 3.7 Cuda: 11.2 Tensorflow: 2.4.1
Code
Relevant API: https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/data/experimental/sample_from_datasets
I cannot provide the full code I use due to it being proprietary, but here is the data loading portion. If more information is needed to root cause this, let me know, and I will see what I can do to provide it. FYI the main code sets all the seeds correctly and disables horovod fusion as suggested by the repo README.
Thanks a lot for the great work on making Tensorflow deterministic. It, along with the documentation provided, has been incredibly useful in my day-to-day work.