`@test_reference` should fail when reference file is missing

[omus]

Calling @test_reference on a non-existent file when JULIA_REFERENCETESTS_UPDATE is unset should cause a test failure instead of generating a new reference file:

julia> @test_reference "dne.txt" "hello"
┌ Info: Reference file for "dne.txt" did not exist. It has been created:
│ - NEW CONTENT -----------------
│ hello
│ -------------------------------
└   new_reference = "/Users/cvogt/.julia/dev/TestReports/dne.txt"
[ Info: Please run the tests again for any changes to take effect

The rational behind this change is that if a user forgets to commit a reference file CI jobs will pass even though the reference test is broken. This happened to me in: https://github.com/JuliaTesting/TestReports.jl/pull/99

[oxinabox]

This is intentional, but perhaps not the wisest feature. It was there to make it easy to initially create the test files, or after a major change recreate all the ones you knew should have changed (by deleting old ones manually then running tests.) With the logic that you could just check the changes in git.

It was added long before https://github.com/JuliaTesting/ReferenceTests.jl/pull/110 Now that we have that, I can see the argument for instead requiring that the user set ENV["JULIA_REFERENCETESTS_UPDATE"]=true but I would say it would be a breaking change, since i know I at least depend on this feature as part of my standard workflow. (but I could learn to change it)

[oxinabox]

as an alternative, possibly we should refuse to create new reference files if ENV["CI"] is true?

[omus]

as an alternative, possibly we should refuse to create new reference files if ENV["CI"] is true?

I think this is a reasonable compromise but would lead to some user confusion in the long term.

[oxinabox]

I think we can just drop automatically creating if missing and the JULIA_REFERENCETESTS_UPDATE enviroment var not set and make this a breaking change -- probably go straight to v1.0.0 with this, since this is the only breaking change we have wanted in years i think we have hit that level of stability.

What do you think @johnnychen94 ?

[johnnychen94]

@oxinabox I'm all good with what you think is good. (Actually, I don't have the bandwidth to proceed with open source projects now 😭)

[oxinabox]

Cool well I am happy to go with doing this.

[oxinabox]

For my own notes this is functionally reverting https://github.com/JuliaTesting/ReferenceTests.jl/pull/52