[WIP] test_reference_broken

[johnnychen94]

Codes here are not ready for review (most of the part is copy&paste).

@test_reference_broken is introduced here, and it works like @test_broken.

---

Some details needs to be clarified and added. But I'd like to show it as a demo of an alternative to #12

A test result:

(ReferenceTests) pkg> test
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Testing ReferenceTests
 Resolving package versions...
[ Info: Nine tests should correctly report failure in the transcript (but not the test summary).
Skipping Base.active_repl
Skipping Base.active_repl_backend
Skipping Base.active_repl
Skipping Base.active_repl_backend
┌ Warning: test fails because PSNR -15.415644645690918 < 25
└ @ ReferenceTests ~/Documents/Julia/ReferenceTests.jl/src/equality_metrics.jl:45
┌ Warning: test fails because size(ref) (64, 64) != size(x) (512, 512)
└ @ ReferenceTests ~/Documents/Julia/ReferenceTests.jl/src/equality_metrics.jl:38
Test Summary:  | Pass  Broken  Total
ReferenceTests |   31       9     40
   Testing ReferenceTests tests passed 

[coveralls]

Coverage decreased (-15.9%) to 66.385% when pulling efdd67c14e4840977bd28b872f4402f739cc154d on johnnychen94:broken into 46d650a2efc73177bc9df7408a579c78488c6849 on Evizero:master.

[oxinabox]

I wonder if we shoukld infact switch to having a function that returns true or false, or throws an error, depending.

Then we could do:

@test matches_reference(actual, "reference.txt")
@test_broken matches_reference(actual, "reference.txt")
@test_skip matches_reference(actual, "reference.txt")
@test !matches_reference(actual, "reference.txt")

rather than having to recrate each.

Are we getting value out of having a top-level macro that we couldn't get out of a function?

[johnnychen94]

I wonder if we shoukld infact switch to having a function that returns true or false, or throws an error, depending.

I like this idea a lot; I've been continuing thinking of replacing @test_reference by test_reference these days.

So export test_reference and deprecate @test_reference?

[oxinabox]

yeah, i think so. We might want to bike-shed on the name a bit first though.

@test test_reference(actual, "reference.txt")

to me doesn't feel right.

and

@test !test_reference(actual, "reference.txt")

feels really wrong. (It is a weird thing to do in the first place but anyway...)

Definately will want a deprecation this time. Which means first tagging a nonbreaking relese with the deprecation then a breaking release (at some point) with the deprecation removed.