Confusing LoadError when loading packages from a non-instantiated manifest that overlaps with a stacked manifest

IanButterworth commented 2 years ago

In the example below, if a package is present in and installed from a parent stacked env (v1.8), and also in a active non-instantiated manifest (DataScience), but something about the version or versions of its deps is different, the error message given when loading seems confusing or just wrong. Conversely, without the overlap with the stacked env the error message and suggestion to instantiate is very clear and helpful.

It seems like a side-effect of https://github.com/JuliaLang/julia/pull/27932 ? https://github.com/JuliaLang/julia/pull/42822 was an attempt to adapt the error message to be more helpful, but I think the error path may just be wrong.

% git clone https://github.com/JuliaAcademy/DataScience

% JULIA_DEPOT_PATH=/tmp/123 julia

(v1.8) pkg> add Images
...

(v1.8) pkg> activate /path/to/DataScience

julia> using Images
[ Info: Precompiling Images [916415d5-f1e6-5110-898d-aaa5f9f070e0]
ERROR: LoadError: ArgumentError: Package SpecialFunctions does not have IrrationalConstants in its dependencies:
- If you have SpecialFunctions checked out for development and have
  added IrrationalConstants as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with SpecialFunctions
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1025 [inlined]
 [2] macro expansion
   @ ./lock.jl:228 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:994
 [4] include
   @ ./Base.jl:416 [inlined]
 [5] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1373
 [6] top-level scope
   @ none:1
 [7] eval
   @ ./boot.jl:368 [inlined]
 [8] eval(x::Expr)
   @ Base.MainInclude ./client.jl:453
 [9] top-level scope
   @ none:1
in expression starting at /tmp/123/packages/SpecialFunctions/NBIqR/src/SpecialFunctions.jl:1
ERROR: LoadError: Failed to precompile SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] to /tmp/123/compiled/v1.8/SpecialFunctions/jl_4iOQOF.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1521
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1465
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1173
  [5] macro expansion
    @ ./loading.jl:1060 [inlined]
  [6] macro expansion
    @ ./lock.jl:228 [inlined]
  [7] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1058
  [8] macro expansion
    @ ./loading.jl:1042 [inlined]
  [9] macro expansion
    @ ./lock.jl:228 [inlined]
 [10] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:994
 [11] include
    @ ./Base.jl:416 [inlined]
 [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1373
 [13] top-level scope
    @ none:1
 [14] eval
    @ ./boot.jl:368 [inlined]
 [15] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [16] top-level scope
    @ none:1
in expression starting at /tmp/123/packages/ColorVectorSpace/0t0Xz/src/ColorVectorSpace.jl:1
ERROR: LoadError: Failed to precompile ColorVectorSpace [c3611d14-8923-5661-9e6a-0046d554d3a4] to /tmp/123/compiled/v1.8/ColorVectorSpace/jl_7czJAq.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1521
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1465
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1173
  [5] macro expansion
    @ ./loading.jl:1060 [inlined]
  [6] macro expansion
    @ ./lock.jl:228 [inlined]
  [7] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1058
  [8] macro expansion
    @ ./loading.jl:1042 [inlined]
  [9] macro expansion
    @ ./lock.jl:228 [inlined]
 [10] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:994
 [11] include
    @ ./Base.jl:416 [inlined]
 [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1373
 [13] top-level scope
    @ none:1
 [14] eval
    @ ./boot.jl:368 [inlined]
 [15] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [16] top-level scope
    @ none:1
in expression starting at /tmp/123/packages/Images/g4e1T/src/Images.jl:3
ERROR: Failed to precompile Images [916415d5-f1e6-5110-898d-aaa5f9f070e0] to /tmp/123/compiled/v1.8/Images/jl_eKP2ev.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1521
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1465
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1173
  [5] macro expansion
    @ ./loading.jl:1060 [inlined]
  [6] macro expansion
    @ ./lock.jl:228 [inlined]
  [7] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1058
  [8] macro expansion
    @ ./loading.jl:1042 [inlined]
  [9] macro expansion
    @ ./lock.jl:228 [inlined]
 [10] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:994

If you then remove Images from the v1.8 environment the error message seems to give the correct error and advice

(DataScience) pkg> activate
  Activating project at `/tmp/123/environments/v1.8`

(v1.8) pkg> st
Status `/private/tmp/123/environments/v1.8/Project.toml`
  [916415d5] Images v0.24.1
  [295af30f] Revise v3.1.20

(v1.8) pkg> rm Images
...

(v1.8) pkg> activate /Users/ian/Documents/GitHub/DataScience
  Activating project at `~/Documents/GitHub/DataScience`

julia> using Images
[ Info: Precompiling Images [916415d5-f1e6-5110-898d-aaa5f9f070e0]
ERROR: LoadError: ArgumentError: Package StaticArrays [90137ffa-7385-5640-81b9-e52037218182] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1142
  [2] macro expansion
    @ ./loading.jl:1060 [inlined]
  [3] macro expansion
    @ ./lock.jl:228 [inlined]
  [4] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1058
  [5] macro expansion
    @ ./loading.jl:1042 [inlined]
  [6] macro expansion
    @ ./lock.jl:228 [inlined]
  [7] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:994
  [8] include
    @ ./Base.jl:416 [inlined]
  [9] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1373
 [10] top-level scope
    @ none:1
 [11] eval
    @ ./boot.jl:368 [inlined]
 [12] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [13] top-level scope
    @ none:1
in expression starting at /tmp/123/packages/Images/g4e1T/src/Images.jl:3
ERROR: Failed to precompile Images [916415d5-f1e6-5110-898d-aaa5f9f070e0] to /tmp/123/compiled/v1.8/Images/jl_nFfWIN.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1521
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1465
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1173
  [5] macro expansion
    @ ./loading.jl:1060 [inlined]
  [6] macro expansion
    @ ./lock.jl:228 [inlined]
  [7] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1058
  [8] macro expansion
    @ ./loading.jl:1042 [inlined]
  [9] macro expansion
    @ ./lock.jl:228 [inlined]
 [10] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:994

IanButterworth commented 2 years ago

AFAICT the two situations that hit this error...

dev work where a package is in an env up the load stack but not yet resolved in the package being developed (the frustration that https://github.com/JuliaLang/julia/pull/27932 appeased)
where the user isn't dev-ing the package, but has yet to instantiate the environment, like the tutorial env example above

...are indistinguishable. The only thing that might distinguish them would be if dev-ed packages were marked with a flag, but even then there are other ways that packages are actively worked on.

So I think that makes the error message addition in https://github.com/JuliaLang/julia/pull/42822 a reasonable improvement here

julia> using Images
[ Info: Precompiling Images [916415d5-f1e6-5110-898d-aaa5f9f070e0]
ERROR: LoadError: ArgumentError: Package SpecialFunctions does not have IrrationalConstants in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- If you have SpecialFunctions checked out for development and have
  added IrrationalConstants as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with SpecialFunctions

IanButterworth commented 2 years ago

Closed by https://github.com/JuliaLang/julia/pull/42822

JuliaLang / julia

Confusing LoadError when loading packages from a non-instantiated manifest that overlaps with a stacked manifest #42873