search

NVIDIA / framework-reproducibility

Providing reproducibility in deep learning frameworks
Apache License 2.0
423 stars 40 forks source link

Are there disadvantages to enabling deterministic ops? #10

Closed lminer closed 4 years ago

lminer commented 4 years ago

Is there any downside to enabling this? Are we losing any speed optimizations for example?

duncanriach commented 4 years ago

Hi @lminer,

Speed would be the only potential disadvantage that I can think of. I have plans to run more extensive performance comparisons on standard models in the future, but the basic performance tests that I have run, on one particular model, showed only a 6% slow down with deterministic ops enabled.

It would be great if you could try training your model with and without and let me know how much slower (or faster) it trains. Also, confirm that you get determinism.

In the longer-term, I do have plans to increase the performance of determinism in TensorFlow, but I cannot commit to when that will be focused on, tackled, and completed.

duncanriach commented 4 years ago

This question has been answered. Closing.