invenia / Models.jl

An interface package that defines the methods and types for working with models.
MIT License
3 stars 0 forks source link

test_interface breaks testsets #15

Closed sdl1 closed 4 years ago

sdl1 commented 4 years ago

E.g. on Julia 1.4:

julia> using Models, Models.TestUtils, Test

# Normal behaviour: test summary is printed, and nothing is assigned to results1
julia> results1 = @testset "test" begin
           @test false
       end
test: Test Failed at REPL[2]:2
  Expression: false
Stacktrace:
 [1] top-level scope at REPL[2]:2
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at REPL[2]:2
Test Summary: | Fail  Total
test          |    1      1
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.

julia> results1.anynonpass
ERROR: UndefVarError: results1 not defined

# Incorrect behaviour: no test summary printed, and returns a DefaultTestSet
# indicating that tests were passed (anynonpass = false)
julia> results2 = @testset "test" begin
           test_interface(FakeTemplate{PointEstimate, SingleOutput}())
           @test false
       end
Models API Interface Test: FakeTemplate: Test Failed at REPL[5]:3
  Expression: false
Stacktrace:
 [1] top-level scope at REPL[5]:3
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [3] top-level scope at REPL[5]:2
Test.DefaultTestSet("test", Any[Test.DefaultTestSet(#= circular reference @-2 =#)], 0, false)

julia> results2.anynonpass
false

Maybe related to https://github.com/JuliaLang/julia/issues/32937?

Seems to be solved by removing return from this line

nickrobinson251 commented 4 years ago

that is an excellent bit of detective work