tupl-tufts / rdl

Types, type checking, and contracts for Ruby
BSD 3-Clause "New" or "Revised" License
602 stars 38 forks source link

At least one test is order dependant #54

Closed ptarjan closed 5 years ago

ptarjan commented 7 years ago

Sharing classes between test cases is dangerous and will lead to spurious test failures. I'd suggest moving to a model where each test's fixtures are run in a separate process so they can't conflict with each other.

Here was my test run. Sadly you don't use the minitest plugin which prints out the random seed so I can't give an exact repro case:

$ rake
warning: parser/current is loading parser/ruby24, which recognizes
warning: 2.4.0-compliant syntax, but you are running 2.4.1.
warning: please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
/Users/pt/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/parser-2.4.0.0/lib/parser/lexer.rb:10922: warning: assigned but unused variable - testEof
/Users/pt/stripe/rdl/lib/rdl/wrap.rb:655: warning: method redefined; discarding old method_added
/Users/pt/stripe/rdl/test/test_typecheck.rb:1704: warning: possibly useless use of :: in void context
/Users/pt/stripe/rdl/test/test_typecheck.rb:1714: warning: possibly useless use of :: in void context
/Users/pt/stripe/rdl/test/test_typecheck.rb:1721: warning: possibly useless use of :: in void context
Run options: --seed 38378

# Running:

.....S......................................................SS...SSSS.......F........................................................S...............................S............

Finished in 0.565990s, 314.4932 runs/s, 2459.4074 assertions/s.

  1) Failure:
TestTypecheck#test_new [/Users/pt/stripe/rdl/test/test_typecheck.rb:929]:
Expected false to be truthy.

178 runs, 1392 assertions, 1 failures, 0 errors, 9 skips

You have skipped tests. Run with --verbose for details.
rake aborted!
Command failed with status (1)

Tasks: TOP => default => test
(See full trace by running task with --trace)
rmosolgo commented 7 years ago

Run options: --seed 38378

Is that the seed for the run in your example above? Could it be used to debug the order-dependency?

ngsankha commented 5 years ago

Fixed with #91.