search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.54k stars 5.47k forks source link

External Pkg always recompiles upon loading #54940

Closed KristofferC closed 2 months ago

KristofferC commented 3 months ago

While being in a Pkg.jl repo:

❯ julia +nightly --project -q

julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: wrong source (1), mismatched flags (1))

❯ julia +nightly --project -q
julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

❯ julia +nightly --project -q
julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

The two original misses are from the bundled cache files so those are expected. This is also a bit weird:

julia> pkg = Base.identify_package("Pkg")
Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]

julia> Base.isprecompiled(pkg)
true

julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))
vchuravy commented 3 months ago

What does JULIA_DEBUG=loading say?

KristofferC commented 3 months ago 
┌ Debug: Rejecting cache file /home/kc/.julia/compiled/v1.12/Pkg/tUTdb_sLFSB.ji because required dependency Base.PkgId(Base.UUID("2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"), "Base64") failed to load from cache file for /home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Base64/D7K0n_Ip9tx.ji.
│   exception = Error reading package image file.

weird..

KristofferC commented 3 months ago

I am also surprised this doesn't print anything at all for Base64:

julia> ENV["JULIA_DEBUG"] = "loading"
"loading"

julia> using Base64

julia> using Example
┌ Debug: Loading object cache file /home/kc/.julia/compiled/v1.12/Example/lLvWP_sLFSB.so for Example [7876af07-990d-54b4-ab0e-23690620f79a]
└ @ Base loading.jl:1227
KristofferC commented 3 months ago

I had some thought that the nightly build juliaup uses was corrupt somehow but building a local Julia on master it gives the exact same behavior.

IanButterworth commented 3 months ago

Base64 is an indirect dep of REPL, so it's already loaded?

KristofferC commented 3 months ago

This method of loading Pkg seems to work:

❯ julia +nightly --project -iqe 'using REPL; using Pkg'
julia> pkgdir(Pkg)
"/home/kc/JuliaPkgs/Pkg.jl"
KristofferC commented 3 months ago

Base64 is an indirect dep of REPL, so it's already loaded?

Okay, that makes sense, I looked in Base.loaded_modules and it wasn't there so I assumed it wasn't loaded already:

julia> Base.loaded_modules
Dict{Base.PkgId, Module} with 12 entries:
  PkgId(UUID("4536629a-c528-5b80-bd46-f80d51c5b363"), "OpenBLAS_jll")          => OpenBLAS_jll
  PkgId(UUID("8f399da3-3557-5675-b5ff-fb832c97cbdb"), "Libdl")                 => Libdl
  PkgId(UUID("ea8e919c-243c-51af-8825-aaa63cd721ce"), "SHA")                   => SHA
  PkgId(UUID("7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"), "FileWatching")          => FileWatching
  PkgId(UUID("56f22d72-fd6d-98f1-02f0-08ddc0907c33"), "Artifacts")             => Artifacts
  PkgId(UUID("8e850b90-86db-534c-a0d3-1478176c7d93"), "libblastrampoline_jll") => libblastrampoline_jll
  PkgId(nothing, "Main")                                                       => Main
  PkgId(nothing, "Core")                                                       => Core
  PkgId(UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra")         => LinearAlgebra
  PkgId(nothing, "Base")                                                       => Base
  PkgId(UUID("9a3f8284-a2c9-5f02-9a11-845980a1fd5c"), "Random")                => Random
  PkgId(UUID("6462fe0b-24de-5631-8697-dd941f90decc"), "Sockets")               => Sockets

so I guess things that are loaded as part of require_stdlib are now invisible but when you do using on them they grab the invisible module and makes it visible?

But that shouldn't make Pkg have to compile over and over due to error when reading the Base64 pkgimage file I guess.

fatteneder commented 3 months ago 
┌ Debug: Rejecting cache file /home/kc/.julia/compiled/v1.12/Pkg/tUTdb_sLFSB.ji because required dependency Base.PkgId(Base.UUID("2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"), "Base64") failed to load from cache file for /home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Base64/D7K0n_Ip9tx.ji.
│   exception = Error reading package image file.

weird..

I have seen this too when trying to debug https://github.com/JuliaPy/PythonCall.jl/issues/513. There @kshyatt nailed down the problem to https://github.com/JuliaLang/julia/commit/71fa11f0427fc66f2328cddbba865852fa47e0f1.

KristofferC commented 3 months ago

There @kshyatt nailed down the problem to https://github.com/JuliaLang/julia/commit/71fa11f0427fc66f2328cddbba865852fa47e0f1.

Same thing happens when I revert that commit.

kshyatt commented 3 months ago

I ran a git bisect for the function I needed not being exported, and it fingered this commit. Maybe multiple things are wrong?

On Wed, Jun 26, 2024 at 9:26 AM Kristoffer Carlsson < @.***> wrote:

There @kshyatt https://github.com/kshyatt nailed down the problem to 71fa11f https://github.com/JuliaLang/julia/commit/71fa11f0427fc66f2328cddbba865852fa47e0f1 .

Same thing happens when I revert that commit.

— Reply to this email directly, view it on GitHub https://github.com/JuliaLang/julia/issues/54940#issuecomment-2191699506, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGKJY5FPJRFDYJSRJHFI4TZJK6P5AVCNFSM6AAAAABJ5V7TROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGY4TSNJQGY . You are receiving this because you were mentioned.Message ID: @.***>

KristofferC commented 3 months ago

Yes, I think they are unrelated.

KristofferC commented 3 months ago

Same happens with OhMyREPL on nightly:

┌ Debug: Loading object cache file /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.dylib for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1227
┌ Debug: Rejecting cache file /Users/kristoffercarlsson/.julia/compiled/v1.12/OhMyREPL/08e1i_J2Ppx.ji because required dependency Base.PkgId(Base.UUID("f489334b-da3d-4c2e-b8f0-e476e12c162b"), "StyledStrings") failed to load from cache file for /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.ji.
│   exception = Error reading package image file.
└ @ Base loading.jl:1962
┌ Debug: Loading object cache file /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.dylib for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1227
┌ Debug: Rejecting cache file /Users/kristoffercarlsson/.julia/compiled/v1.12/OhMyREPL/08e1i_J2Ppx.ji because required dependency Base.PkgId(Base.UUID("f489334b-da3d-4c2e-b8f0-e476e12c162b"), "StyledStrings") failed to load from cache file for /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.ji.
│   exception = Error reading package image file.