Open evolu8 opened 3 years ago
Sorry for the delay in responding, Phil; I was on vacation.
I have not run this code or got into debugging it. Just from looking at it, I can see a couple of likely sources of nondeterminism:
tf.keras.layers.BatchNormalization
is instantiated in five places in models.py
. This layer uses fused batch-norm functionality, which is nondeterministic when being used for fine-tuning. I don't know under exactly what circumstances that is exposed by the Keras layer and, since I wasn't aware of this exposure until now, I have yet to documented it.tf.nn.sparse_softmax_cross_entropy_with_logits
is used in linearevaluation.py
on the output of the ClassificationHead. This op will introduce nondeterminism, and there is a work-around for it.Answering your specific questions/comments:
Running TF 2.4.1 with seeds and envs set I'm getting different results each run for this guy:
https://github.com/garder14/byol-tensorflow2
I currently suspect it's the gradient tape. Not sure how to handle that. Would downgrading TF version help?
Thoughts welcome.