hunse
commented
5 years ago Dan had the idea that setting the seed could solve it, since it could be some irregular float32 rounding that happens more with particular seeds. Once we can reliably reproduce it, this might be the first thing to try. https://github.com/nengo/nengo-dl/pull/87#issuecomment-485816657
When I encountered this, it failed on TravisCI (Python 3.6 build, no environment variables set). I re-ran the build, and it passed, so definitely something non-deterministic going on here. I ran the test 10 times on my own machine and didn't get a failure, so either a) it only fails very occasionally, or b) the failures are specific to the TravisCI setup. My guess is (a).
I think the first step is to run this ~100 times locally and see if we can reproduce it. Probably something in the simulator causing things to be non-deterministic.