search

neuroailab / tfutils

Utilities for working with tensorflow
MIT License
25 stars 8 forks source link

Results are not stable, even with fixed seed #15

Open yamins81 opened 7 years ago

yamins81 commented 7 years ago

I've noticed that output results are not stable, meaning that different runs produce the different results, even with the same initial conditions. Specifically, even with all random seeds held constant, and with the FIFO queue, although initial filter values at t = 0 are identical between runs, they have diverged by t = 1. I haven't been able isolate the source of the divergence, including whether it's the result of numerical error.

qbilius commented 7 years ago

Could it be due to us running train and validation queues at the same time? Maybe they get mixed up somehow? Have you tried disabling validation?