search

NVIDIA / framework-reproducibility

Providing reproducibility in deep learning frameworks
Apache License 2.0
423 stars 40 forks source link

Support for TF 2.1.0 #8

Closed jmwoloso closed 4 years ago

jmwoloso commented 4 years ago

First of all, thank you so much for this package! I'm using databricks and they've upgraded some of their runtimes to TF 2.1.0 and was wondering if I might request support for that. It isn't a blocker for me (I just downgrade to 2.0.0), but would be nice to have moving forward. Thanks for your work on this!

duncanriach commented 4 years ago

Thanks for the appreciation, @jmwoloso. You're welcome.

Good news! I have upstreamed changes into TensorFlow 2.1 so that it doesn't require patching. Instead of using patch(), you can enable determinism the way it's currently done for the NGC TF containers. I'm sorry that this will require a change to your code.

I have not yet updated the tensorflow-determinism documentation to reflect this, but I plan to do that soon.

To prevent future user-side code changes as the configuration of determinism in TensorFlow changes, I plan to add a new function to the package called something like enable_determinism which will work on any version of TensorFlow and will handle the nuances of what needs to be done for any given version to maximally enable determinism, possibly executing recipe components beyond just patching. Stay tuned.

jmwoloso commented 4 years ago

Thanks a million @duncanriach! My code should actually be fine as I already set os.environ["TF_DETERMINISTIC_OPS"] = '1' in my code in addition to using patch(). Small price to pay for reliably reproducible results in TF. I'd be happy to tackle the docs if you need it. Thanks again!

duncanriach commented 4 years ago

You're welcome, @jmwoloso! Thanks for offering to update the docs, and I already committed the necessary changes yesterday.