search

JuliaLang / Pkg.jl

Pkg - Package manager for the Julia programming language
https://pkgdocs.julialang.org
Other
625 stars 265 forks source link

Resolving 1.11 manifest in 1.10 makes Statistics "not downloaded" #4058

Open fonsp opened 3 hours ago

fonsp commented 3 hours ago

I created an environment in Julia 1.11 (see attached zip), and then I resolved it in Julia 1.10. This worked, but "Statistics" shows as "not downloaded", and instantiate cannot fix it.

Overview of what happened:

(testenv11) pkg> resolve

(testenv11) pkg> st -m

...
→ [10745b16] Statistics v1.10.0
...

Info Packages marked with → are not downloaded, use `instantiate` to download

(testenv11) pkg> instantiate
  0 dependencies successfully precompiled in 0 seconds. 17 already precompiled. 3 skipped during auto due to previous errors.

(testenv11) pkg> precompile
...

ERROR: LoadError: ArgumentError: Package Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Full log

Full reproducer (with empty julia depot):

Project+Manifest to start with (from Julia 1.11): julia 1.11 env original.zip

Project+Manifest at the end of this terminal session (from Julia 1.10) with broken Statistics: testenv11.zip

➜  ~ JULIA_DEPOT_PATH="/tmp/asdfff" julia +1.10 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.5 (2024-08-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(v1.10) pkg> activate Downloads/testenv11/
  Activating project at `~/Downloads/testenv11`

(testenv11) pkg> st -m
  Installing known registries into `/tmp/asdfff`
       Added `General` registry to /tmp/asdfff/registries
Status `~/Downloads/testenv11/Manifest.toml`
→  [6e696c72] AbstractPlutoDingetjes v1.3.2
→⌅ [3da002f7] ColorTypes v0.11.5
→  [53c48c17] FixedPointNumbers v0.8.5
→  [47d2ed2b] Hyperscript v0.0.5
→  [ac1192a8] HypertextLiteral v0.9.5
→  [b5f81e59] IOCapture v0.2.5
→  [682c06a0] JSON v0.21.4
→  [6c6e2e6c] MIMEs v0.1.4
→  [69de0a69] Parsers v2.8.1
→  [7f904dfe] PlutoUI v0.7.60
→  [aea7be01] PrecompileTools v1.2.1
→  [21216c6a] Preferences v1.4.3
→  [189a3867] Reexport v1.2.2
→  [410a4b4d] Tricks v0.1.9
→  [5c2747f8] URIs v1.5.1
   [0dad84c5] ArgTools v1.1.2
   [56f22d72] Artifacts v1.11.0
   [2a0f44e3] Base64 v1.11.0
   [ade2ca70] Dates v1.11.0
   [f43a241f] Downloads v1.6.0
   [7b1f6079] FileWatching v1.11.0
   [b77e0a4c] InteractiveUtils v1.11.0
   [b27032c2] LibCURL v0.6.4
   [76f85450] LibGit2 v1.11.0
   [8f399da3] Libdl v1.11.0
   [37e2e46d] LinearAlgebra v1.11.0
   [56ddb016] Logging v1.11.0
   [d6f4376e] Markdown v1.11.0
   [a63ad114] Mmap v1.11.0
   [ca575930] NetworkOptions v1.2.0
   [44cfe95a] Pkg v1.11.0
   [de0858da] Printf v1.11.0
   [9a3f8284] Random v1.11.0
   [ea8e919c] SHA v0.7.0
   [9e88b42a] Serialization v1.11.0
→  [10745b16] Statistics v1.11.1
   [fa267f1f] TOML v1.0.3
   [a4e569a6] Tar v1.10.0
   [8dfed614] Test v1.11.0
   [cf7118a7] UUIDs v1.11.0
   [4ec0a83e] Unicode v1.11.0
   [e66e0078] CompilerSupportLibraries_jll v1.1.1+0
   [deac9b47] LibCURL_jll v8.6.0+0
   [e37daf67] LibGit2_jll v1.7.2+0
   [29816b5a] LibSSH2_jll v1.11.0+1
   [c8ffd9c3] MbedTLS_jll v2.28.6+0
   [14a3606d] MozillaCACerts_jll v2023.12.12
   [4536629a] OpenBLAS_jll v0.3.27+1
   [83775a58] Zlib_jll v1.2.13+1
   [8e850b90] libblastrampoline_jll v5.11.0+0
   [8e850ede] nghttp2_jll v1.59.0+0
   [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with → are not downloaded, use `instantiate` to download
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Warning The project dependencies or compat requirements have changed since the manifest was last resolved. It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.

(testenv11) pkg> resolve
   Installed Tricks ───────────────── v0.1.9
   Installed JSON ─────────────────── v0.21.4
   Installed IOCapture ────────────── v0.2.5
   Installed Hyperscript ──────────── v0.0.5
   Installed Parsers ──────────────── v2.8.1
   Installed FixedPointNumbers ────── v0.8.5
   Installed Preferences ──────────── v1.4.3
   Installed ColorTypes ───────────── v0.11.5
   Installed PrecompileTools ──────── v1.2.1
   Installed URIs ─────────────────── v1.5.1
   Installed Reexport ─────────────── v1.2.2
   Installed MIMEs ────────────────── v0.1.4
   Installed HypertextLiteral ─────── v0.9.5
   Installed AbstractPlutoDingetjes ─ v1.3.2
   Installed PlutoUI ──────────────── v0.7.60
  No Changes to `~/Downloads/testenv11/Project.toml`
    Updating `~/Downloads/testenv11/Manifest.toml`
  [0dad84c5] ↓ ArgTools v1.1.2 ⇒ v1.1.1
  [56f22d72] ~ Artifacts v1.11.0 ⇒ 
  [2a0f44e3] ~ Base64 v1.11.0 ⇒ 
  [ade2ca70] ~ Dates v1.11.0 ⇒ 
  [7b1f6079] ~ FileWatching v1.11.0 ⇒ 
  [b77e0a4c] ~ InteractiveUtils v1.11.0 ⇒ 
  [76f85450] ~ LibGit2 v1.11.0 ⇒ 
  [8f399da3] ~ Libdl v1.11.0 ⇒ 
  [37e2e46d] ~ LinearAlgebra v1.11.0 ⇒ 
  [56ddb016] ~ Logging v1.11.0 ⇒ 
  [d6f4376e] ~ Markdown v1.11.0 ⇒ 
  [a63ad114] ~ Mmap v1.11.0 ⇒ 
  [44cfe95a] ↓ Pkg v1.11.0 ⇒ v1.10.0
  [de0858da] ~ Printf v1.11.0 ⇒ 
  [3fa0cd96] + REPL
  [9a3f8284] ~ Random v1.11.0 ⇒ 
  [9e88b42a] ~ Serialization v1.11.0 ⇒ 
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays v1.10.0
→ [10745b16] ↓ Statistics v1.11.1 ⇒ v1.10.0
  [8dfed614] ~ Test v1.11.0 ⇒ 
  [cf7118a7] ~ UUIDs v1.11.0 ⇒ 
  [4ec0a83e] ~ Unicode v1.11.0 ⇒ 
  [deac9b47] ↓ LibCURL_jll v8.6.0+0 ⇒ v8.4.0+0
  [e37daf67] ↓ LibGit2_jll v1.7.2+0 ⇒ v1.6.4+0
  [c8ffd9c3] ↓ MbedTLS_jll v2.28.6+0 ⇒ v2.28.2+1
  [14a3606d] ↓ MozillaCACerts_jll v2023.12.12 ⇒ v2023.1.10
  [4536629a] ↓ OpenBLAS_jll v0.3.27+1 ⇒ v0.3.23+4
  [bea87d4a] + SuiteSparse_jll v7.2.1+1
  [8e850ede] ↓ nghttp2_jll v1.59.0+0 ⇒ v1.52.0+1
        Info Packages marked with → are not downloaded, use `instantiate` to download

(testenv11) pkg> instantiate
Precompiling project...
  ✗ FixedPointNumbers
  ✗ ColorTypes
  ✗ PlutoUI
  17 dependencies successfully precompiled in 14 seconds
  4 dependencies errored.
  For a report of the errors see `julia> err`. To retry use `pkg> precompile`

julia> err
PkgPrecompileError: The following 1 direct dependency failed to precompile:

PlutoUI [7f904dfe-b85e-4ff6-b463-dae2292396a8]

Failed to precompile PlutoUI [7f904dfe-b85e-4ff6-b463-dae2292396a8] to "/tmp/asdfff/compiled/v1.10/PlutoUI/jl_aH6cHB".
ERROR: LoadError: ArgumentError: Package Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1926
  [2] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
  [3] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
  [4] invoke_in_world
    @ ./essentials.jl:923 [inlined]
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
  [6] macro expansion
    @ ./loading.jl:1790 [inlined]
  [7] macro expansion
    @ ./lock.jl:267 [inlined]
  [8] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
  [9] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [10] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [12] include
    @ ./Base.jl:495 [inlined]
 [13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
 [14] top-level scope
    @ stdin:3
in expression starting at /tmp/asdfff/packages/FixedPointNumbers/Dn4hv/src/FixedPointNumbers.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile FixedPointNumbers [53c48c17-4a7d-5ca2-90c5-79b7896eea93] to "/tmp/asdfff/compiled/v1.10/FixedPointNumbers/jl_rTMnck".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include
    @ ./Base.jl:495 [inlined]
 [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
 [25] top-level scope
    @ stdin:3
in expression starting at /tmp/asdfff/packages/ColorTypes/vpFgh/src/ColorTypes.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile ColorTypes [3da002f7-5984-5a60-b8a6-cbb66c0b333f] to "/tmp/asdfff/compiled/v1.10/ColorTypes/jl_qoteAT".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.5+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include(mod::Module, _path::String)
    @ Base ./Base.jl:495
 [24] include(x::String)
    @ PlutoUI.BuiltinsNotebook /tmp/asdfff/packages/PlutoUI/SzOYa/src/PlutoUI.jl:11
 [25] top-level scope
    @ /tmp/asdfff/packages/PlutoUI/SzOYa/src/PlutoUI.jl:12
 [26] include
    @ ./Base.jl:495 [inlined]
 [27] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2222
 [28] top-level scope
    @ stdin:3
in expression starting at /tmp/asdfff/packages/PlutoUI/SzOYa/src/Builtins.jl:1060
in expression starting at /tmp/asdfff/packages/PlutoUI/SzOYa/src/PlutoUI.jl:1
in expression starting at stdin:

(testenv11) pkg> st -m
Status `~/Downloads/testenv11/Manifest.toml`
  [6e696c72] AbstractPlutoDingetjes v1.3.2
⌅ [3da002f7] ColorTypes v0.11.5
  [53c48c17] FixedPointNumbers v0.8.5
  [47d2ed2b] Hyperscript v0.0.5
  [ac1192a8] HypertextLiteral v0.9.5
  [b5f81e59] IOCapture v0.2.5
  [682c06a0] JSON v0.21.4
  [6c6e2e6c] MIMEs v0.1.4
  [69de0a69] Parsers v2.8.1
  [7f904dfe] PlutoUI v0.7.60
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [189a3867] Reexport v1.2.2
  [410a4b4d] Tricks v0.1.9
  [5c2747f8] URIs v1.5.1
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [b77e0a4c] InteractiveUtils
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
→ [10745b16] Statistics v1.10.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.1+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.11.0+0
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with → are not downloaded, use `instantiate` to download
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`

(testenv11) pkg> instantiate
  0 dependencies successfully precompiled in 0 seconds. 17 already precompiled. 3 skipped during auto due to previous errors.

(testenv11) pkg>
IanButterworth commented 3 hours ago

You should have seen a warning about the manifest having been resolved for a different julia version?

fonsp commented 2 hours ago

Sorry I shared the wrong zip file, fixed now.

fonsp commented 2 hours ago

@IanButterworth Thanks for taking a look! The warning is missing indeed, but I mainly want to point out:

With resolve on Julia 1.10, Statistics 1.11 was downgraded to 1.10 (good), and it showed a message that it is not downloaded and that instantiate will download it. But instantiate did not download Statistics 1.10. And it looks like you are left in a "broken state".

So I think something went wrong with the resolve.

fonsp commented 2 hours ago

In the Manifest.toml that was generated in my issue (where Statistics could not load), I have the following:

julia_version = "1.10.5"

...

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
git-tree-sha1 = "ae3bb1eb3bba077cd276bc5cfc337cc65c3075c0"
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
version = "1.10.0"

But if I create a new env in Julia 1.10 and add Statistics, the Manifest.toml contains:

julia_version = "1.10.5"

...

[[deps.Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
version = "1.10.0"
IanButterworth commented 2 hours ago

I guess this is related to Statistics being an upgradable stdlib (here confusingly named "former stdlib") https://github.com/JuliaLang/Pkg.jl/blob/116ba910c74ab565d348aa8a50d6dd10148f11ab/src/Types.jl#L463

IanButterworth commented 2 hours ago

Do you have the manifest entry for the 1.11 manifest? I wonder if the git-tree-sha1 isn't updated.

fonsp commented 2 hours ago

From the original 1.11 manifest:

[[deps.Statistics]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "ae3bb1eb3bba077cd276bc5cfc337cc65c3075c0"
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
version = "1.11.1"

    [deps.Statistics.extensions]
    SparseArraysExt = ["SparseArrays"]

    [deps.Statistics.weakdeps]
    SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
IanButterworth commented 1 hour ago

Yeah, the git-tree-sha1 should have changed. 1.10 is on a different Statistics commit 1.11 https://github.com/JuliaLang/julia/blob/release-1.11/stdlib/Statistics.version

1.10 https://github.com/JuliaLang/julia/blob/release-1.10/stdlib/Statistics.version