Closed gavins13 closed 4 years ago
Hey @gavinlive!
Thanks for tracking down and reporting this source of non-determinism. I want to capture this knowledge into the document, but I don't believe that Confirmed Current GPU-Specific Sources of Non-Determinism (With Solutions) is the correct location. Without thoroughly exploring the underlying cause, it's not clear to me that this is about a purely GPU-related source of non-determinism. Do you agree with that assessment? if so, I think that it should be covered in two places:
I also think that this information can be made more general. The use of tf.while_loop
when parallel_iterations
is greater than 1 (noting that 10 is the default) may introduce non-determinism into model functionality. Then, additionally, the AutoGraph Transformations caused by tf.function
may lead to loops being implemented using tf.while_loop
and therefore parallelized, which may introduce non-determinism into model functionality.
Please will you enhance this PR appropriately?
Finally, I would like to add your name to the credits. I can simply add @gavinlive, if you want, or if you give me your first and last name then I will add that.
Hey @gavinlive!
Thanks for tracking down and reporting this source of non-determinism. I want to capture this knowledge into the document, but I don't believe that Confirmed Current GPU-Specific Sources of Non-Determinism (With Solutions) is the correct location. Without thoroughly exploring the underlying cause, it's not clear to me that this is about a purely GPU-related source of non-determinism. Do you agree with that assessment? if so, I think that it should be covered in two places:
- Additional Ingredients in the Determinism Recipe
- Sources of Non-Determinism in TensorFlow Unrelated to GPU
I also think that this information can be made more general. The use of
tf.while_loop
whenparallel_iterations
is greater than 1 (noting that 10 is the default) may introduce non-determinism into model functionality. Then, additionally, the AutoGraph Transformations caused bytf.function
may lead to loops being implemented usingtf.while_loop
and therefore parallelized, which may introduce non-determinism into model functionality.Please will you enhance this PR appropriately?
Finally, I would like to add your name to the credits. I can simply add @gavinlive, if you want, or if you give me your first and last name then I will add that.
Hi @duncanriach,
I will enhance this PR. You can add me as Gavin Seegoolam, thank you!
Oh, hey Gavin! We interacted by email recently. Did you get my response to your email?
Oh, hey Gavin! We interacted by email recently. Did you get my response to your email?
Yes, I received you response. I believe I responded to it but just in case, basically I added a new TensorFlow issue here: https://github.com/tensorflow/tensorflow/issues/39751 and also added another pull request to this repo which references this.
Hope you're doing well!
I didn't receive an email response from you, but I'll get to that other pull request, and the TF issue, soon.
Hey @gavinlive, I want to get this PR merged before it starts conflicting. I'm planning to merge it and then do the enhancements I mentioned above.
Included solution for AutoGraph's conversion of
for
loops towhile_loops
with parallel iterations on the GPU