search

racket / htdp

Other
91 stars 69 forks source link

random tests fail occasionally #132

Closed samth closed 3 years ago

samth commented 3 years ago

Some of the randomized tests in tests/test-engine/racket-tests fail occasionally. See http://drdr.racket-lang.org/56554/racket/share/pkgs/htdp-test/tests/test-engine/racket-tests.rkt for an example.

The relevant failing tests are:

I'm not certain what the specification being tested here is, or I would propose a fix.

The code was originally added in https://github.com/racket/htdp/commit/26e36b6dade7d6c7c95ea1c019e4a7808c82a8ed and may actually be by @dvanhorn.

mfelleisen commented 3 years ago

DVH suggested this addition, with sketches of tests. I implemented them and left DVH’s suggestion in as comments. It is safe to comment them out (so they stay as an acknowledgement of his inspiration).

dvanhorn commented 3 years ago

I believe the idea was that check-random sets up the rng seed so that a sequence of calls to random in the checked expression will be the same as a sequence of calls in the expected expressions (assuming the same arguments).

It looks like these tests which have different sequences of calls (and therefore intended to fail) are occasionally getting (un)lucky and producing the same results.

I think it would be fine to comment out, or you could make the sequence longer so that this kind of coincidence is astronomically unlikely (and we should buy lottery tickets when the tests fail).

David

On Tue, Feb 9, 2021 at 5:45 PM Matthias Felleisen notifications@github.com wrote:

DVH suggested this addition, with sketches of tests. I implemented them and left DVH’s suggestion in as comments. It is safe to comment them out (so they stay as an acknowledgement of his inspiration).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/racket/htdp/issues/132#issuecomment-776294893, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABSEZMEXETTOW7XGK6A6R3S6G3IDANCNFSM4XLMX7QA .

samth commented 3 years ago

I like the idea of adding more checks, that should fix the problem nicely, especially if the numbers are larger.