search

NVIDIA / framework-reproducibility

Providing reproducibility in deep learning frameworks
Apache License 2.0
423 stars 40 forks source link

How can I check whether I'm in a NGC TensorFlow container? #6

Closed AndreaPi closed 4 years ago

AndreaPi commented 4 years ago

As it is today, tensorflow-determinism is activated in two different ways, whether we're in a NGC Tensorflow Docker container > 19.06, or we're running stock Tensorflow. This begs the question: how do I check in which of the two enviroments my code is running? I need to write code which is deterministic, no matter whether the user is using Docker or not. Thus, I need to be able to check at runtime if I'm using the Tensorflow version which comes with NGC, or the stock one. How do I do that?

duncanriach commented 4 years ago

Very clear question. Thank you.

I recommend using the same mechanism that the patch uses to error check and prevent the patch being applied inside the NGC TF container.

As an aside, I have also considered (in a later version) allowing the patch to be applied to an NGC TF container (>= 19.06), in which case it will simply set TF_DETERMINISTIC_OPS=true. Your use-case adds to my motivation to do this.

duncanriach commented 4 years ago

Update to my comment about about allowing the patch to be applied to an NGC TF container: my current plan is to add a function to the package called something like enable_determinism which will be callable on any version of TensorFlow, whether inside an NGC TF container or not, and it will make the maximum possible effort to enable determinism in that version of TensorFlow.

This question has been answered. Closing.