search

ajschumacher / ajschumacher.github.io

blog
http://planspace.org/
20 stars 21 forks source link

randomness in neural nets #259

Open ajschumacher opened 3 years ago

ajschumacher commented 3 years ago

Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance https://www.cs.purdue.edu/homes/lintan/publications/variance-ase20.pdf https://dl.acm.org/doi/10.1145/3324884.3416545 December 2020

A Workaround for Non-Determinism in TensorFlow https://www.twosigma.com/articles/a-workaround-for-non-determinism-in-tensorflow/

Determinism in Deep Learning (S9911) Duncan Riach, GTC 2019 https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9911-determinism-in-deep-learning.pdf

The funny thing still is that even if you make your runs deterministic, you haven't eliminated randomness, you've just frozen a particular random example.

Underspecification Presents Challenges for Credibility in Modern Machine Learning https://arxiv.org/abs/2011.03395

ajschumacher commented 3 years ago

TF_CUDNN_DETERMINISTIC, TF_CUDNN_USE_AUTOTUNE