search

JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
82 stars 14 forks source link

It should be possible to regenerate $n$ test files without restarting the tests $n$ times #3

Closed oxinabox closed 6 years ago

oxinabox commented 6 years ago

When I change the output of my function to produce a different string, then I know that all of my reference tests are going to fail.

So I delete my 5 test files.

Now, to recreate them I have to rerun julia 5 times. because after recreating the test files it throws an error.

I think instead of throwing an error it should do something the testset can catch. Like maybe an @test false or an @test_broken false ?

Evizero commented 6 years ago

I know exactly what you are saying just by reading the title. I have encountered this before. I agree this should be changed. I actually thought I made the change already

I think this can be fixed by putting lines like this in a warn instead of error https://github.com/Evizero/ReferenceTests.jl/blob/master/src/test_reference.jl#L108 . maybe with a @test false, but I think failed tests also abort the test sequence

oxinabox commented 6 years ago

This is really annoying and is still happening.