Closed louislbc closed 4 years ago
Hi @louislbc, I am surprised to hear of tf.keras.layers.Conv2D
injecting non-determinism when running on a GPU, even when upsampling. I have made a note to attempt to repro that. Luckily, I am currently working on a solution for tf.image.resize
(with the default method=ResizeMethod.BILINEAR
). That solution should be released via the NGC TF container version 20.03 (available in March) before being upstreamed into stock TensorFlow.
Also, please will you let me know if the new title does not match what you're looking for. I assume that you're using this upsampling in a trainable model which relies on the back-prop through the upsampling up.
Sorry for the late reply and thanks for your great work. A typo in the question and I have corrected it. I found tf.keras.layers.UpSampling2D and tf.image.resize cannot ensure deterministic and back-propagatable upsampling. I used Conv2DTranspose instead to perform upsampling as an alternative. But I hope tf.image.resize would soon be released to replace it.
Thanks for the follow-up, @louislbc. tf.keras.layers.UpSampling2D
uses tf.image.resize
, so both of these are exhibiting the same issue. Thanks for sharing that Conv2DTranspose
can be used as a work-around. That matches what I would expect.
In TF 2.x, tf.image.resize
with method=ResizeMethod.BILINEAR
, which is the default, will have deterministic back-prop in NGC (NVIDIA GPU Cloud) TensorFlow container version 20.03
, releasing in March 2020. I will also be upstreaming this solution to stock TensorFlow as soon as possible. I'm not sure if I will be able to get this into stock TF 2.2, but it will most likely be in stock TF 2.3. For others who might be reading this, and who want to continue using the TF1 API, this feature will also be available in the TF1 version of the container, via tf.image.resize_bilinear
Deterministic back-prop withtf.keras.layers.UpSampling2D
, will then also be available when interpolation='bilinear'
is selected.
Hey @louislbc, Could I get your full name to add to the credits in this repo?
I found when performing upsampling, both tf.keras.layers.UpSampling2D and tf.image.resize can not ensure determinism in my code (tensorflow2.0, ubuntu 16.04, python 3.7, cuda10.0, cudnn7.6) . Is there any setting or altenative mothod to perform upsampling that ensures determinism? Thank you!