Crashing on Julia 1.11-rc1

[marcelo-g-simas]

When I try to use TestReports.jl on Julia 1.11.0-rc1 (2024-06-25) I get a crash before any tests run. Here's what happens:

julia> using TestReports; TestReports.test("WesVAR")
[ Info: Testing WesVAR
ERROR: KeyError: key "REPL" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:477 [inlined]
  [2] Base.Precompilation.ExplicitEnv(envpath::String)
    @ Base.Precompilation ./precompilation.jl:201
  [3] Base.Precompilation.ExplicitEnv()
    @ Base.Precompilation ./precompilation.jl:29
  [4] precompilepkgs(pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, timing::Bool, _from_loading::Bool, configs::Pair{Cmd, Base.CacheFlags}, io::Base.TTY, fancyprint::Bool)
    @ Base.Precompilation ./precompilation.jl:367
  [5] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2377
  [6] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2216
  [7] #invoke_in_world#3
    @ ./essentials.jl:1077 [inlined]
  [8] invoke_in_world
    @ ./essentials.jl:1074 [inlined]
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2207
 [10] macro expansion
    @ ./loading.jl:2146 [inlined]
 [11] macro expansion
    @ ./lock.jl:273 [inlined]
 [12] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2103
 [13] #invoke_in_world#3
    @ ./essentials.jl:1077 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1074 [inlined]
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2096
ERROR: TestReports failed to generate the report.
See error log above.

[mharradon]

I'm also running into this issue on julia 1.11.0.

[KristofferC]

TestReports.jl generates an invalid manifest when it creates the test manifest that it uses. The issue is:

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
version = "1.11.0"
weakdeps = ["REPL"]

    [deps.Pkg.extensions]
    REPLExt = "REPL"

If there is no [[deps.REPL]] in the manifest this weak dependency needs to be saved in "expanded form" (which includes the UUID) as this:

[[deps.Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
version = "1.11.0"

    [deps.Pkg.extensions]
    REPLExt = "REPL"

    [deps.Pkg.weakdeps]
    REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

The Pkg project writer handles this but this package instead chooses to directly print the manifest dictionary which circumvents that.

[mmiller-max]

Hey, please can you see if this is fixed on the latest version (v"1.2.0")? Now switched to using TestEnv which has been updated for 1.11. Thanks

[mharradon]

I'm getting a similar error still:

Precompiling project...
    877.3 ms  ✓ CoverageTools
   1197.4 ms  ✓ Coverage
  2 dependencies successfully precompiled in 2 seconds. 367 already precompiled.
   Resolving package versions...
   Installed EzXML ─────── v1.2.0
   Installed TestEnv ───── v1.102.0
   Installed TestReports ─ v1.2.0
      Compat entries added for TestReports
    Updating `/builds/project/project/Project.toml`
  [dcd651b4] + TestReports v1.2.0
    Updating `/builds/project/project/Manifest.toml`
  [8f5d6c58] + EzXML v1.2.0
  [1e6cf692] + TestEnv v1.102.0
  [dcd651b4] + TestReports v1.2.0
Precompiling project...
   1259.9 ms  ✓ EzXML
   1394.7 ms  ✓ TestEnv
   1867.7 ms  ✓ TestReports
  3 dependencies successfully precompiled in 4 seconds. 369 already precompiled.
[ Info: Testing PACKAGE
ERROR: KeyError: key "REPL" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:477 [inlined]
  [2] Base.Precompilation.ExplicitEnv(envpath::String)
    @ Base.Precompilation ./precompilation.jl:201
  [3] Base.Precompilation.ExplicitEnv()
    @ Base.Precompilation ./precompilation.jl:29
  [4] precompilepkgs(pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, timing::Bool, _from_loading::Bool, configs::Pair{Cmd, Base.CacheFlags}, io::Base.PipeEndpoint, fancyprint::Bool)
    @ Base.Precompilation ./precompilation.jl:367
  [5] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2466
  [6] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2300
  [7] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
  [8] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2287
 [10] macro expansion
    @ ./loading.jl:2226 [inlined]
 [11] macro expansion
    @ ./lock.jl:273 [inlined]
 [12] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2183
 [13] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
 [14] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2176
ERROR: TestReports failed to generate the report.
See error log above.

[KristofferC]

The issue is in this function https://github.com/JuliaTesting/TestReports.jl/blob/ef6dd386318dc71815d778b8f52cbff40c363d76/src/runner.jl#L78-L109.

It creates TOML dictionaries and prints them but fails to uphold the requirements of a proper manifest (https://github.com/JuliaTesting/TestReports.jl/issues/121#issuecomment-2413243321).