search

google / objax

Apache License 2.0
769 stars 77 forks source link

Random number tests fail unpredictably #124

Closed carlini closed 3 years ago

carlini commented 3 years ago

There are currently tests in tests/testrandom.py that fail with small probability (and some with not-so-small probability)

For example, here

https://github.com/google/objax/blob/07b2bb09980ee92935973015ea10d1fa99e5e766/tests/testrandom.py#L54-L55

the probability that this fails is ~11% (taking the mean of 1000*100 samples with man stdev=2 should be distributed like a normal with mean 006.324e-3 and this is greater than .01 with probability 11%).

The other tests also fail with non-negligible probability.

david-berthelot commented 3 years ago

Would setting a fix seed per test address be a satisfactory solution?

carlini commented 3 years ago

Does jax guarantee that the same seed will generate the same random sequence across minor versions of the library? If so, then maybe.

The seed should be reset before every random call though, or else changing one test could make a new one fail.

david-berthelot commented 3 years ago

Well we can pass a generator per test so the seed can be controlled locally without trouble. As to JAX guarantees, I don't know.

carlini commented 3 years ago

I think we can close this now. @david-berthelot