nick8325 / quickcheck

Automatic testing of Haskell programs.
Other
713 stars 119 forks source link

Quickcheck seed determined by system time in seconds. #316

Closed PeterRugg closed 5 months ago

PeterRugg commented 3 years ago

We run several instances of QuickCheck in parallel. Recently we noticed that the parallel runs were all producing exactly the same counterexample, and on further investigations, exactly the same tests. Between versions 2.12.something and 2.13.2 this issue materialised, and is still present in 2.14.1. It seems that the default RNG (with GHC) has switched to SMGen, which seeds its RNG with the current POSIX time, in seconds! I would argue that this is not expected behaviour, so ideally QuickCheck would override the seed, or at the very least warn users in the documentation not to run multiple instances in parallel without paying attention to seeding. It could also be argued that this is an issue with SMGen so should be bumped further up the chain.

phadej commented 3 years ago

newQCGen uses newSMGen

https://github.com/nick8325/quickcheck/blob/73c676f6451a27aff04ce60b20b9ffa8f34faf28/src/Test/QuickCheck/Random.hs#L64-L69

See https://hackage.haskell.org/package/splitmix-0.1.0.1/docs/System-Random-SplitMix.html

-- | Initialize SMGen using system time.
initSMGen :: IO SMGen

-- | Derive a new generator instance from the global SMGen using splitSMGen.
newSMGen :: IO SMGen

I suspect that you call initSMGen somewhere, when you should call newSMGen. Or you use completely separate processes.


I do plan to make initSMGen to use some other sources of randomness (like process id), but unfortunately base doesn't expose anything like that so I'd need to write some C / JS code (Windows, GHCJS).

(I could hack some solution to read e.g. /dev/urandom on Linux, like tf-random does but I'd like to have proper solution on Windows and GHCJS too).

PeterRugg commented 3 years ago

Indeed, we do use completely separate processes. It's possible that makes this a somewhat unusual use-case.

Agreed that it would be nice for it to be cross-platform. I suppose that it really is SM Gen's or our problem to solve. Just got a little taken aback when different users were seeing/not seeing the issue when using different versions of QuickCheck. I'll watch that issue closely. Thanks for the quick response!

phadej commented 3 years ago

The https://hackage.haskell.org/package/splitmix-0.1.0.2 have initial seeds affected by process id