`Pkg.test` should maybe instantiate the full test project

[Keno]

Consider the following

$ find -type f -exec batcat --style header {} +
File: ./test/runtests.jl
using SciMLSensitivity

File: ./docs/Project.toml
[deps]
SciMLSensitivity = "1ed8b502-d754-442c-8d5d-10ac956f44a1"

[sources]
SciMLSensitivity = { url = "https://github.com/Keno/SciMLSensitivity.jl", rev = "kf/mindep" }

File: ./Project.toml
name = "PkgTestWeirdness"
uuid = "78ed732d-4f3c-41ca-8ab7-3ddcca345aff"

[workspace]
projects = ["docs"]

[extras]
SciMLSensitivity = "1ed8b502-d754-442c-8d5d-10ac956f44a1"

[targets]
test = ["SciMLSensitivity"]

Then at the REPL

(PkgTestWeirdness) pkg> resolve

shell> rm -rf ~/.julia/packages/SciMLSensitivity/ ; simulate moving to CI

(PkgTestWeirdness) pkg> test
     Testing PkgTestWeirdness
ERROR: expected package `SciMLSensitivity [1ed8b502]` to exist at path `/home/keno/.julia/packages/SciMLSensitivity/IpVTR`

Ordinary instantiate (as both CI and Pkg.test run) does not fix this:

(PkgTestWeirdness) pkg> instantiate
  0 dependencies successfully precompiled in 0 seconds
  1 dependency errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

(PkgTestWeirdness) pkg> test
     Testing PkgTestWeirdness
ERROR: expected package `SciMLSensitivity [1ed8b502]` to exist at path `/home/keno/.julia/packages/SciMLSensitivity/IpVTR`

instantiate --workspace does fix this, but it's not what CI runs (though it could).

Possible solutions I can see:

1. The Pkg.instantiate that Pkg.test does should instantiate the full test project, not just the toplevel project. The issues here is that this breaks the case where SciMLSensitivity is not already in the manifest.

2. The resolve that Pkg.test does should maybe ignore the fact that the manifest is non-instantiated. The pkg is pinned and the toplevel manifest should have sufficient information for it to do everything that it needs. Then, we could instantiate after the resolve, which would avoid the previously mentioned problem.

[KristofferC]

Probably has the same root cause as https://github.com/JuliaLang/Pkg.jl/issues/860 (hello issue from 6 years ago). Also https://github.com/JuliaLang/Pkg.jl/issues/3859. Basically, resolve errors if a package added via URL is not downloaded. Can be reprod easier via:

(TestInst) pkg> st
Project TestInst v0.1.0
Status `~/JuliaTests/TestInst/Project.toml`
→ [1ed8b502] SciMLSensitivity v7.56.2 `https://github.com/Keno/SciMLSensitivity.jl#kf/mindep`
Info Packages marked with → are not downloaded, use `instantiate` to download

(TestInst) pkg> resolve
ERROR: expected package `SciMLSensitivity [1ed8b502]` to exist at path `/Users/kristoffercarlsson/.julia/packages/SciMLSensitivity/IpVTR`

[Keno]

Your PR fixes two of the tree ways I tried, but this not the third one. Reproducer:

$ find -type f -exec batcat --style header {} +
File: ./test/runtests.jl
using SciMLSensitivity

File: ./test/Project.toml
[deps]
SciMLSensitivity = "1ed8b502-d754-442c-8d5d-10ac956f44a1"

File: ./docs/Project.toml
[deps]
SciMLSensitivity = "1ed8b502-d754-442c-8d5d-10ac956f44a1"

[sources]
SciMLSensitivity = { url = "https://github.com/Keno/SciMLSensitivity.jl", rev = "kf/mindep" }

File: ./Project.toml
name = "PkgTestWeirdness"
uuid = "78ed732d-4f3c-41ca-8ab7-3ddcca345aff"

[workspace]
projects = ["docs", "test"]

File: ./src/PkgTestWeirdness.jl
module PkgTestWeirdness
end

Same setup script and then:

(PkgTestWeirdness) pkg> test
[snip]
     Testing Running tests...
ERROR: LoadError: ArgumentError: Package SciMLSensitivity [1ed8b502-d754-442c-8d5d-10ac956f44a1] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.