include("test/runtests.jl") does not work with extra dependencies

[juliohm]

This is an example where extra dependencies aren't visible: https://github.com/JuliaImages/ImageInpainting.jl

ReferenceTests.jl returns an error saying that the dependencies aren't available in the environment. Should I split the Project.toml of the package into two separate Project.toml, one for the package and one for tests? Is this already working with the latest Pkg.jl?

[oxinabox]

What Julia version?

[juliohm]

I am in Julia v1.6-beta1 and the GitHub Actions are running on the latest stable so Julia v1.5.

[johnnychen94]

What do you mean by "extra dependencies"? It would be much helpful if there's a log to diagnose.

[juliohm]

I mean the section of the Project.toml that is only available during test time, not an actual dependency of the project: https://github.com/JuliaImages/ImageInpainting.jl/blob/f6bddbf9b6b0ef16ceb0edae4991751a835b7ade/Project.toml#L19-L25

The command ] test Foo seems to magically install these extra packages when we have the package environment activated. Simply doing include("test/runtests.jl") doesn't work because these extra packages aren't available. Makse sense?

[oxinabox]

Correct. include("test/runtests.jl") does not activate the testing enviroment. This is https://github.com/JuliaLang/Pkg.jl/issues/1233

one thing you can do is delete the reference files that should have changed. and then run ] test. Any missing reference files will be created. I have often deleted all of my reference files before running the tests. then using git diff tools to check that only the ones that should have changed did change.

[juliohm]

Couldn't we use a similar approach of Documenter.jl where a separate Project.toml/Manifest.toml is used for the test suite? Is this what the issue linked is proposing?

Something along the lines of:

; cd test
] activate .
] add ..
] add extra1 extra2 ...

Now we have a separate Project.toml in the test folder pointing to the dev version of the package.

[oxinabox]

Out of scope for this package

[johnnychen94]

According to https://julialang.github.io/Pkg.jl/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above, test/Project.toml is specially handled, so probably you could indeed use it in a way like docs/Project.toml while still making >pkg test working.

For compatibility reason, I haven't tried this approach yet so I don't know if it fits the routine perfectly. But as @oxinabox said, this is a Pkg issue and is out of the scope of this package; RefereneceTests is just a normal Julia package and has no special requirements on Pkg.