Strange behavior with existing manifest and missing project

fredrikekre commented 3 years ago

I verified that I did not have a project file before, but I guess from the output that I must have had a manifest. On add Crayons, why was JSON@0.19.0 installed in the first place leading to a failure later where it was (rightfully) missing from the manifest?

$ pkg --project=. add Crayons
  Resolving package versions...
  Installed JSON ─ v0.19.0
Updating `~/Project.toml`
  [a8cc5b0e] + Crayons v4.0.4
Updating `~/Manifest.toml`
  [a8cc5b0e] + Crayons v4.0.4
  [fb57b044] - Foo v0.1.0 `Foo`
  [682c06a0] - JSON v0.19.0
  [2a0f44e3] - Base64
  [ade2ca70] - Dates
  [8ba89e20] - Distributed
  [b77e0a4c] - InteractiveUtils
  [76f85450] - LibGit2
  [8f399da3] - Libdl
  [37e2e46d] - LinearAlgebra
  [56ddb016] - Logging
  [d6f4376e] - Markdown
  [a63ad114] - Mmap
  [44cfe95a] - Pkg
  [de0858da] - Printf
  [3fa0cd96] - REPL
  [9a3f8284] - Random
  [ea8e919c] - SHA
  [9e88b42a] - Serialization
  [6462fe0b] - Sockets
  [8dfed614] - Test
  [cf7118a7] - UUIDs
  [4ec0a83e] - Unicode
ERROR: LoadError: could not find entry with uuid 682c06a0-de6a-54ab-a142-c8b1cf79cde6 in manifest /home/fredrik/Manifest.toml
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] build_versions(::Pkg.Types.Context, ::Array{Base.UUID,1}; might_need_to_resolve::Bool, verbose::Bool) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:898

fredrikekre commented 3 years ago

Reproducer:

$ cd $(mktemp -d)

$ export JULIA_DEPOT_PATH=$PWD

$ export JULIA_LOAD_PATH=$PWD/Project.toml

$ jlpkg add JSON &> /dev/null 

$ rm -rf packages/JSON Project.toml

$ jlpkg add Crayons
  Resolving package versions...
  Installed Crayons ─ v4.0.4
  Installed JSON ──── v0.21.1
Updating `/tmp/tmp.NKv4yx8IEc/Project.toml`
  [a8cc5b0e] + Crayons v4.0.4
Updating `/tmp/tmp.NKv4yx8IEc/Manifest.toml`
  [a8cc5b0e] + Crayons v4.0.4
  [682c06a0] - JSON v0.21.1
  [69de0a69] - Parsers v1.0.12
  [ade2ca70] - Dates
  [a63ad114] - Mmap
  [de0858da] - Printf
  [4ec0a83e] - Unicode
ERROR: LoadError: could not find entry with uuid 682c06a0-de6a-54ab-a142-c8b1cf79cde6 in manifest /tmp/tmp.NKv4yx8IEc/Manifest.toml
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] build_versions(::Pkg.Types.Context, ::Array{Base.UUID,1}; might_need_to_resolve::Bool, verbose::Bool) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:898
 [3] build_versions at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:888 [inlined]
 [4] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Array{Base.UUID,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Linux) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:1145
 [5] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.Linux, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:188
 [6] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:139
 [7] add(::Array{Pkg.Types.PackageSpec,1}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:67
 [8] add(::Array{Pkg.Types.PackageSpec,1}) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:67
 [9] do_cmd!(::Pkg.REPLMode.Command, ::Pkg.REPLMode.MiniREPL) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:404
 [10] do_cmd(::Pkg.REPLMode.MiniREPL, ::String; do_rethrow::Bool) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:382
 [11] pkgstr(::String) at /home/fredrik/julia15/usr/share/julia/stdlib/v1.5/Pkg/src/REPLMode/REPLMode.jl:464
 [12] top-level scope at /usr/local/bin/jlpkg:192
 [13] include(::Function, ::Module, ::String) at ./Base.jl:380
 [14] include(::Module, ::String) at ./Base.jl:368
 [15] exec_options(::Base.JLOptions) at ./client.jl:296
 [16] _start() at ./client.jl:506
in expression starting at /usr/local/bin/jlpkg:191

KristofferC commented 3 years ago

So what I think happens:

JSON is in the manifest
The tiered resolver tries to fix all the packages in the manifest and resolve.
This causes JSON to get installed (even though it is not a dep anything in the project).
The manifest is pruned, removing the JSON entry from the manifest.
The build step runs, notices that JSON should be built because it was installed but it is no longer in the manifest

This should be fixable by just pruning the manifest on the creation of the EnvCache.

The second problem is the whole Vector{::PackageSpec} we have going around everywhere. When this gets desynced with the environment, weird things happen. I have some local work that only uses PackageSpec as an input to the resolver step, which returns a new environment and then the environment is used everywhere, but that is quite a bit away from PR-ing.

JuliaLang / Pkg.jl

Strange behavior with existing manifest and missing project #2244