search

Seelengrab / Supposition.jl

A Julia implementation of choice sequence based PBT, inspired by Hypothesis
https://seelengrab.github.io/Supposition.jl/
European Union Public License 1.2
46 stars 2 forks source link

[Bug]: Test report table does not display properly #53

Closed sztal closed 1 month ago

sztal commented 1 month ago

What happened?

Hi!

The problem as in the title. Below is is the minimal example for reproducing the error.

I basically run the code from the README and get unformatted output.

julia> using Test, Supposition

julia> @testset "Examples" begin

           # Get a generator for `Int8`
           intgen = Data.Integers{Int8}()

           # Define a property `foo` and feed it `Int8` from that generator
           @check function foo(i=intgen)
               i isa Int
           end

           # Define & run another property, reusing the generator
           @check function bar(i=intgen)
               i isa Int8
           end

           # Define a property that can error
           @check function baba(i=intgen)
               i < -5 || error()
           end

           # Feed a new generator to an existing property
           @check bar(Data.Floats{Float16}())

           # Mark properties as broken
           @check broken=true function broke(b=Data.Booleans())
               b isa String
           end

           # ...and lots more, so check out the docs!
       end

The final part of the output, instead of being a nice table like in the README, looks more or less like this:

Test.DefaultTestSet("Examples", Any[Supposition.SuppositionReport("foo", "foo(Int8)", Some(Supposition.TestState(Supposition.CheckConfig(Random.Xoshiro(0x20389cd18ea3b24f, 0x079dc621e515b347, 0xc51eb37eef442178, 0xb999adf6ebeb3f21, 0x23df32725da82d63), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710), var"##foo__run#336#16"{var"##foo__geninput#335#15"{Supposition.Data.Integers{Int8, UInt8}}}(Core.Box(var"#foo#17"()), var"##foo__geninput#335#15"{Supposition.Data.Integers{Int8, UInt8}}(Supposition.Data.Integers{Int8}())), Random.Xoshiro(0x20389cd18ea3b24f, 0x079dc621e515b347, 0xc51eb37eef442178, 0xb999adf6ebeb3f21, 0x23df32725da82d63), 0x0000000000000001, 0x0000000000000004, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x0000000000000000], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])), nothing, nothing, Tuple{Type, Base.StackTraces.StackFrame}[], false, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x0000000000000000], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])))), Some(Supposition.Fail((i = -128,), Pair{AbstractString, Any}[], nothing)), 1.72720397206335e9, Some(1.727203972147333e9), Supposition.CheckConfig(Random.Xoshiro(0x20389cd18ea3b24f, 0x079dc621e515b347, 0xc51eb37eef442178, 0xb999adf6ebeb3f21, 0x23df32725da82d63), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710)), Supposition.SuppositionReport("bar", "bar(Int8)", Some(Supposition.TestState(Supposition.CheckConfig(Random.Xoshiro(0x6d8a9e767c9b0ad8, 0xb4afc091aac754b5, 0xf675cf8d1aac4c7b, 0xa0e9156dfae2804a, 0xc24683eede800964), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710), var"##bar__run#358#19"{var"##bar__geninput#357#18"{Supposition.Data.Integers{Int8, UInt8}}}(Core.Box(var"#bar#20"()), var"##bar__geninput#357#18"{Supposition.Data.Integers{Int8, UInt8}}(Supposition.Data.Integers{Int8}())), Random.Xoshiro(0xf021b29fbfb13a3f, 0xd45f0a88ac9905b0, 0x00aed043095b4fa9, 0x3aa09519886bc42b, 0xc24683eede800964), 0x0000000000002710, 0x0000000000002710, nothing, nothing, nothing, Tuple{Type, Base.StackTraces.StackFrame}[], false, nothing)), Some(Supposition.Pass(nothing, Pair{AbstractString, Any}[], nothing)), 1.72720397215078e9, Some(1.727203972208622e9), Supposition.CheckConfig(Random.Xoshiro(0x6d8a9e767c9b0ad8, 0xb4afc091aac754b5, 0xf675cf8d1aac4c7b, 0xa0e9156dfae2804a, 0xc24683eede800964), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710)), Supposition.SuppositionReport("baba", "baba(Int8)", Some(Supposition.TestState(Supposition.CheckConfig(Random.Xoshiro(0x507a60ec5636163c, 0x2185afa035868541, 0x8c039246e5a7c912, 0x33109df00e7f6fe4, 0xd6919cbfd88ca295), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710), var"##baba__run#380#22"{var"##baba__geninput#379#21"{Supposition.Data.Integers{Int8, UInt8}}}(Core.Box(var"#baba#23"()), var"##baba__geninput#379#21"{Supposition.Data.Integers{Int8, UInt8}}(Supposition.Data.Integers{Int8}())), Random.Xoshiro(0x507a60ec5636163c, 0x2185afa035868541, 0x8c039246e5a7c912, 0x33109df00e7f6fe4, 0xd6919cbfd88ca295), 0x000000000000000d, 0x0000000000000010, nothing, nothing, Some((ErrorException(""), Base.StackTraces.StackFrame[error() at error.jl:44, (::var"#baba#23")(i::Int8) at REPL[7]:18, (::var"##baba__run#380#22"{var"##baba__geninput#379#21"{Supposition.Data.Integers{Int8, UInt8}}})(378::Supposition.TestCase{Random.Xoshiro}) at api.jl:240, macro expansion at teststate.jl:38 [inlined], macro expansion at ScopedValues.jl:198 [inlined], test_function(ts::Supposition.TestState, tc::Supposition.TestCase{Random.Xoshiro}) at teststate.jl:37, generate!(ts::Supposition.TestState) at teststate.jl:317, run(ts::Supposition.TestState) at teststate.jl:187, macro expansion at api.jl:311 [inlined], macro expansion at Test.jl:1577 [inlined]  …  start_repl_backend(backend::REPL.REPLBackend, consumer::Any; get_module::Function) at REPL.jl:231, kwcall(::NamedTuple, ::typeof(REPL.start_repl_backend), backend::REPL.REPLBackend, consumer::Any) at REPL.jl:228, run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool, backend::Any) at REPL.jl:389, run_repl(repl::REPL.AbstractREPL, consumer::Any) at REPL.jl:375, (::Base.var"#1013#1015"{Bool, Bool, Bool})(REPL::Module) at client.jl:432, #invokelatest#2 at essentials.jl:892 [inlined], invokelatest at essentials.jl:889 [inlined], run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool) at client.jl:416, exec_options(opts::Base.JLOptions) at client.jl:333, _start() at client.jl:552], 6, Supposition.Attempt(UInt64[0x0000000000000000, 0x000000000000007b], 0x0000000000000001, 10000, Pair{AbstractString, Any}[]))), Tuple{Type, Base.StackTraces.StackFrame}[], false, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x000000000000007b], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])))), Some(Supposition.Error((i = -5,), Pair{AbstractString, Any}[], ErrorException(""), Base.StackTraces.StackFrame[error() at error.jl:44, (::var"#baba#23")(i::Int8) at REPL[7]:18, (::var"##baba__run#380#22"{var"##baba__geninput#379#21"{Supposition.Data.Integers{Int8, UInt8}}})(378::Supposition.TestCase{Random.Xoshiro}) at api.jl:240, macro expansion at teststate.jl:38 [inlined]])), 1.727203972212513e9, Some(1.727203972776848e9), Supposition.CheckConfig(Random.Xoshiro(0x507a60ec5636163c, 0x2185afa035868541, 0x8c039246e5a7c912, 0x33109df00e7f6fe4, 0xd6919cbfd88ca295), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710)), Supposition.SuppositionReport("bar", "bar(Float16)", Some(Supposition.TestState(Supposition.CheckConfig(Random.Xoshiro(0x02f6a5ad2705608a, 0xe745a6217debf9fb, 0x79ae83cf0de94c4e, 0x481ae59d481f7520, 0x11ec7269df33ffe1), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710), var"##bar__run#402#25"{var"##bar__geninput#401#24"}(var"##bar__geninput#401#24"(), Core.Box(var"#bar#20"())), Random.Xoshiro(0x02f6a5ad2705608a, 0xe745a6217debf9fb, 0x79ae83cf0de94c4e, 0x481ae59d481f7520, 0x11ec7269df33ffe1), 0x0000000000000001, 0x0000000000000004, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x0000000000000000], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])), nothing, nothing, Tuple{Type, Base.StackTraces.StackFrame}[], false, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x0000000000000000], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])))), Some(Supposition.Fail((arg_1 = Float16(0.0),), Pair{AbstractString, Any}[], nothing)), 1.727203972781474e9, Some(1.727203972804861e9), Supposition.CheckConfig(Random.Xoshiro(0x02f6a5ad2705608a, 0xe745a6217debf9fb, 0x79ae83cf0de94c4e, 0x481ae59d481f7520, 0x11ec7269df33ffe1), 10000, true, false, false, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710)), Supposition.SuppositionReport("broke", "broke(Bool)", Some(Supposition.TestState(Supposition.CheckConfig(Random.Xoshiro(0xfa750822d8a0cc2c, 0x3883cff970aaaf50, 0x5656cf7f5927a02a, 0x18f71d8c69cfcbb6, 0x02745462cd158ce8), 10000, true, false, true, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710), var"##broke__run#423#27"{var"##broke__geninput#422#26"}(Core.Box(var"#broke#28"()), var"##broke__geninput#422#26"()), Random.Xoshiro(0xfa750822d8a0cc2c, 0x3883cff970aaaf50, 0x5656cf7f5927a02a, 0x18f71d8c69cfcbb6, 0x02745462cd158ce8), 0x0000000000000001, 0x0000000000000004, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x0000000000000000], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])), nothing, nothing, Tuple{Type, Base.StackTraces.StackFrame}[], false, Some(Supposition.Attempt(UInt64[0x0000000000000000, 0x0000000000000000], 0x0000000000000001, 10000, Pair{AbstractString, Any}[])))), Some(Supposition.Fail((b = false,), Pair{AbstractString, Any}[], nothing)), 1.727203972809888e9, Some(1.727203972826926e9), Supposition.CheckConfig(Random.Xoshiro(0xfa750822d8a0cc2c, 0x3883cff970aaaf50, 0x5656cf7f5927a02a, 0x18f71d8c69cfcbb6, 0x02745462cd158ce8), 10000, true, false, true, Supposition.DirectoryDB("/tmp/jl_6v3bV7/test/SuppositionDB"), 0x0000000000002710))], 0, false, false, true, 1.727203972059636e9, 1.727203972827145e9, false, "REPL[7]")

Am I doing something wrong or is it a bug?

Cheers!

What did you expect to happen?

I expect the output to look like the nice table from the README.

Julia Version

Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 9 5900HX with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)
Environment:
  JULIA_PROJECT = @.
  JULIA_PKG_PRESERVE_TIERED_INSTALLED = true
  JULIA_REVISE = manual

Package Environment

[5a0628fe] Supposition v0.3.5
Seelengrab commented 1 month ago

So, there's good news & bad news about this:

The reason for this is because of https://github.com/JuliaLang/julia/pull/53215 (which I opened precisely due to this issue). There isn't really anything I can do about 1.10, unfortunately.. I could add a note to the README linking that PR?

Note that the only thing that's missing is the summary table (the printing of which is provided by the Test stdlib), the individual printing-of-counterexamples is still in there, it's just pushed down due to the long printing of Test.DefaultTestSet. As a workaround, you can add a trailing ; to the top-level invocation of @testset to suppress the existing show method being called on the return value of @testset.

sztal commented 1 month ago

Ok, thanks a lot for the explanation!

By the way, do you know whether Supposition.jl interoperates correctly with the TestItems.jl framework?

Seelengrab commented 1 month ago

I don't, sorry! I don't use VSCode, and there doesn't seem to be any documentation for how to hook into this.

Since your original issue was answered (and there's nothing I can do about 1.10), I'm going to close the issue :)

sztal commented 1 month ago

Sure, no problem, thanks a lot!