[Preferences] Test ignores preferences that are not explicitly listed as test dependencies.

[carstenbauer]

Came up over at MPI.jl here.

Currently, ] test ignores JULIA_LOAD_PATH. On our cluster, we use the latter to expand the default load path to an extra directory (via JULIA_LOAD_PATH=:/some/global/path) that contains a Project.toml that sets some global preferences (e.g. for MPI.jl). Consequently, the global preferences aren't respected within ] test.

I see two possible ways forward:

1) Make ] test respect JULIA_LOAD_PATH. (Why doesn't it?) 2) Propagate all preferences that are set along the load path to the test environment. (This is along the lines of https://github.com/JuliaLang/Pkg.jl/issues/2500 and https://github.com/JuliaLang/Pkg.jl/pull/2920 which didn't cover the case of global preferences, IIUC.)

(Tested with Julia 1.8.5)

[carstenbauer]

Maybe I'm wrong with my point 2) above. MWE:

Minimal package with the following Project.toml and test/runtest.jl:

name = "mwe"
uuid = "4187ee2f-2d69-45e0-8479-75d21fecdd8d"
authors = ["Carsten Bauer <crstnbr@gmail.com>"]
version = "0.1.0"

[deps]
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Preferences = "21216c6a-2e73-6563-6e65-726566657250"

[extras]
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Preferences = "21216c6a-2e73-6563-6e65-726566657250"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test", "LinearAlgebra", "Preferences"]

@show LOAD_PATH
using Preferences, LinearAlgebra
@show Preferences.load_preference(LinearAlgebra, "testpref")

Now, an extra Project.toml in some global path (say /some/global/path) with the following content:

[extras]
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[preferences.LinearAlgebra]
testpref = 1

Running JULIA_LOAD_PATH=:/some/global/path julia --project and then ] test I get

LOAD_PATH = ["@", "/var/folders/qk/nm34_hqn7_7g8fvxgpg24gm40000gn/T/jl_CemQ7p"]
Preferences.load_preference(LinearAlgebra, "testpref") = 1

So, despite the fact that /some/global/path isn't in LOAD_PATH, the preference seems to propagate fine!

(Why didn't it for MPIPreferences than?)

[vchuravy]

breadcrumbs #3061 and https://github.com/JuliaParallel/MPI.jl/issues/561

cc: @staticfloat

[vchuravy]

What happens if you add MPIPreferences to the test dependencies?

[carstenbauer]

What happens if you add MPIPreferences to the test dependencies?

That works!

So I take it that only preferences of direct test dependencies are forwarded?

[vchuravy]

Hm still smells like a bug, we use Base.get_preferences to collect the Preferences to propagate, and while #3061 added load path, get_preferences might only collect preferences of direct dependencies.

But looking at https://github.com/JuliaLang/julia/blob/b5482c82d486aaa68939871eb1d1bc71bb421096/base/loading.jl#L2608 I don't see how or why that would happen/

[vchuravy]

@staticfloat https://github.com/JuliaLang/Pkg.jl/blob/63e9558b9786556dc92b606d1e3f22ffd0af00af/src/Operations.jl#L1911 returns a dictonary of Name=>Prefs but no associated Name=>UUID mapping. We then copy over the preferences https://github.com/JuliaLang/Pkg.jl/blob/63e9558b9786556dc92b606d1e3f22ffd0af00af/src/Operations.jl#L1742 to the test environment, but preference loading will ignore them since the tmp_project does not contain an appropriate Name=>UUID mapping.