JuliaTesting / ReferenceTests.jl

Utility package for comparing data against reference files
https://juliatesting.github.io/ReferenceTests.jl/latest/
Other
82 stars 14 forks source link

rework `@test_reference` messages - drop lena #118

Closed t-bltg closed 7 months ago

t-bltg commented 1 year ago

As an example, this job output (PR) is more readable than this one (master): most likely flush of stdout and stderr is de-synchronized.

PR

julia> ENV["VERBOSE"] = true  # default

pkg> test ReferenceTests
[...]
Test Summary:    | Pass  Total
world age issues |    1      1
┌ Info: Reference Test for "string1.txt" failed:
│ - DIFF ------------------------
│ "fintentionally wrobng to check that this message prints"
│ -------------------------------
│   reference = "[...]/test/references/string1.txt"
└   actual = "/tmp/jl_h48D23/string1.txt"
┌ Info: Reference Test for "string5.txt" failed:
│ - DIFF ------------------------
│ """
│ - This is a
│ + This is an incorrect
│   multiline string that does not end with a new line."""
│ -------------------------------
│   reference = "[...]/test/references/string5.txt"
└   actual = "/tmp/jl_79MGXW/string5.txt"
[...]
     Testing ReferenceTests tests passed

julia> ENV["VERBOSE"] = false

pkg> test ReferenceTests
[...]
Test Summary:    | Pass  Total
world age issues |    1      1
Test Summary:  | Pass  Total
ReferenceTests |   49     49
     Testing ReferenceTests tests passed 

master

pkg> test ReferenceTests
[...]
┌ Info: Reference Test for "string1.txt" failed.
│   reference = "[...]/test/references/string1.txt"
└   actual = "/tmp/jl_egX948/string1.txt"
- DIFF ------------------------
"fintentionally wrobng to check that this message prints"
-------------------------------
┌ Info: Reference Test for "string5.txt" failed.
│   reference = "[...]/test/references/string5.txt"
└   actual = "/tmp/jl_qicFon/string5.txt"
- DIFF ------------------------
"""
- This is a
+ This is an incorrect
  multiline string that does not end with a new line."""
-------------------------------
[...]
     Testing ReferenceTests tests passed 
t-bltg commented 7 months ago

@oxinabox, I have some issues with nightly CI in UnicodePlots (see e.g. corrupted paths with ANSI codes in https://github.com/JuliaPlots/UnicodePlots.jl/actions/runs/7300585284/job/19895531268#step:6:384).

This PR makes the output from ReferenceTests clearer for debugging.

Can we please revive and merge this PR and bump a new release (maybe @johnnychen94) ?

t-bltg commented 7 months ago

CI is green for all OS.