[WIP] Fail rather than error if can't regenerate

[oxinabox]

I'm not sure this is the right way to test this.

[Evizero]

There must be a simpler way to test this. Maybe a simple @test_warn to check that the warning prints and thus the code is reached.

[oxinabox]

There must be a simpler way to test this. Maybe a simple @test_warn to check that the warning prints and thus the code is reached.

Problem is @test fail bubbles up the testsets and fails this test.

Maybe we can define our own Testset that doesn't propagate failures?

[Evizero]

or @test_broken?

[Evizero]

If @test_broken works, it would at least offer a reference code of how to handle tests that fail without failing

[oxinabox]

@test_broken deals with expressions that are false. Where are we are deal with tests that fail.

Consider:

julia> foo(x) = @test false
foo (generic function with 1 method)

julia> @testset "Test all the things" begin
           @test_broken foo(1)
       end
Test all the things: Test Failed
  Expression: false
Stacktrace:
 [1] foo(::Int64) at ./REPL[47]:1
 [2] macro expansion at ./REPL[49]:2 [inlined]
 [3] macro expansion at ./test.jl:860 [inlined]
 [4] anonymous at ./<missing>:?
Test Summary:       | Fail  Broken  Total
Test all the things |    1       1      2
ERROR: Some tests did not pass: 0 passed, 1 failed, 0 errored, 1 broken.

the @test still gets to trigger the testset's record So now we get a Fail & a Broken