search

NVIDIA / framework-reproducibility

Providing reproducibility in deep learning frameworks
Apache License 2.0
423 stars 40 forks source link

How can I get deterministic, back-propagatable upsampling? #12

Closed louislbc closed 4 years ago

louislbc commented 4 years ago

I found when performing upsampling, both tf.keras.layers.UpSampling2D and tf.image.resize can not ensure determinism in my code (tensorflow2.0, ubuntu 16.04, python 3.7, cuda10.0, cudnn7.6) . Is there any setting or altenative mothod to perform upsampling that ensures determinism? Thank you!

duncanriach commented 4 years ago

Hi @louislbc, I am surprised to hear of tf.keras.layers.Conv2D injecting non-determinism when running on a GPU, even when upsampling. I have made a note to attempt to repro that. Luckily, I am currently working on a solution for tf.image.resize (with the default method=ResizeMethod.BILINEAR). That solution should be released via the NGC TF container version 20.03 (available in March) before being upstreamed into stock TensorFlow.

duncanriach commented 4 years ago

Also, please will you let me know if the new title does not match what you're looking for. I assume that you're using this upsampling in a trainable model which relies on the back-prop through the upsampling up.

louislbc commented 4 years ago

Sorry for the late reply and thanks for your great work. A typo in the question and I have corrected it. I found tf.keras.layers.UpSampling2D and tf.image.resize cannot ensure deterministic and back-propagatable upsampling. I used Conv2DTranspose instead to perform upsampling as an alternative. But I hope tf.image.resize would soon be released to replace it.

duncanriach commented 4 years ago

Thanks for the follow-up, @louislbc. tf.keras.layers.UpSampling2D uses tf.image.resize, so both of these are exhibiting the same issue. Thanks for sharing that Conv2DTranspose can be used as a work-around. That matches what I would expect.

In TF 2.x, tf.image.resize with method=ResizeMethod.BILINEAR, which is the default, will have deterministic back-prop in NGC (NVIDIA GPU Cloud) TensorFlow container version 20.03, releasing in March 2020. I will also be upstreaming this solution to stock TensorFlow as soon as possible. I'm not sure if I will be able to get this into stock TF 2.2, but it will most likely be in stock TF 2.3. For others who might be reading this, and who want to continue using the TF1 API, this feature will also be available in the TF1 version of the container, via tf.image.resize_bilinear

Deterministic back-prop withtf.keras.layers.UpSampling2D, will then also be available when interpolation='bilinear' is selected.

duncanriach commented 4 years ago

Hey @louislbc, Could I get your full name to add to the credits in this repo?