search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.43k stars 5.45k forks source link

precompilation problems with 1.11.0-rc2 #55340

Open matthias314 opened 1 month ago

matthias314 commented 1 month ago

This issue looks similar to #53983, but details are different. On top of that, I don't know how to reproduce it others can reproduce it. Here is a very simple case of the problem:

I have an existing depot in /usr/local/julia. I make sure that there are no subfolders compiled/v1.11 or environments/v1.11 and then do the following:

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
julia> using Base64
[ Info: Precompiling Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] 

julia> exit()

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
julia> using Base64
[ Info: Precompiling Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] (cache misses: wrong source (1))

julia> exit()

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
julia> using Base64
[ Info: Precompiling Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] (cache misses: wrong source (1))

and so on. With a newly created, completely empty depot the problem does not arise. Is there anything else that I should delete inside the existing depot? Any other ideas?

Julia Version 1.11.0-rc2
Commit 34c3a63147b (2024-07-29 06:24 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i3-10110U CPU @ 2.10GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /usr/local/julia
nsajko commented 1 month ago

The "wrong source" message seems to indicate there's some kind of mismatch with some source file. Perhaps you or some program edited a source file? Can you reproduce this after deleting /usr/local/julia-1.11.0-rc2 and extracting the archive again?

vtjnash commented 1 month ago

Try with JULIA_DEBUG=loading in your environment to get more debugging metadata associated with the loading.jl file?

matthias314 commented 1 month ago

@nsajko

Can you reproduce this after deleting /usr/local/julia-1.11.0-rc2 and extracting the archive again?

Yes, it doesn't change everything. I've also verified the SHA256 checksum of the tar file.

@vtjnash This looks promising. I start again with empty /usr/local/julia/compiled/v1.11:

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Base64/D7K0n_G9ii0.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Markdown/AREjX_G9ii0.so for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/InteractiveUtils/0TrXF_G9ii0.so for InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/StyledStrings/UcVoM_G9ii0.so for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Unicode/E4Hzs_G9ii0.so for Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/REPL/u0gqU_G9ii0.so for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1185
julia> using Base64
[ Info: Precompiling Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] 
┌ Debug: Generating object cache file for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:2805
┌ Debug: Loading object cache file /usr/local/julia/compiled/v1.11/Base64/D7K0n_iafuA.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185

julia> exit()

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Base64/D7K0n_G9ii0.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Markdown/AREjX_G9ii0.so for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/InteractiveUtils/0TrXF_G9ii0.so for InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/StyledStrings/UcVoM_G9ii0.so for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Unicode/E4Hzs_G9ii0.so for Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/REPL/u0gqU_G9ii0.so for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1185
julia> using Base64
┌ Debug: Rejecting cache file /usr/local/julia/compiled/v1.11/Base64/D7K0n_iafuA.ji because it is for file @depot-1.11.0-rc2/share/julia/stdlib/v1.11/Base64/src/Base64.jl not file /usr/local/julia-1.11.0-rc2/share/julia/stdlib/v1.11/Base64/src/Base64.jl
└ @ Base loading.jl:3713
┌ Debug: Rejecting cache file /usr/local/julia/compiled/v1.11/Base64/D7K0n_iafuA.ji because it is for file @depot-1.11.0-rc2/share/julia/stdlib/v1.11/Base64/src/Base64.jl not file /usr/local/julia-1.11.0-rc2/share/julia/stdlib/v1.11/Base64/src/Base64.jl
└ @ Base loading.jl:3713
[ Info: Precompiling Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] (cache misses: wrong source (1))
┌ Debug: Generating object cache file for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:2805
┌ Debug: Loading object cache file /usr/local/julia/compiled/v1.11/Base64/D7K0n_iafuA.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185

Note the @depot-1.11.0-rc2 on the second run. I've also noticed that on the first run Base64 is first loaded from /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Base64/D7K0n_G9ii0.so and then after saying using Base64 from /usr/local/julia/compiled/v1.11/Base64/D7K0n_iafuA.so.

With a completely empty /tmp/depot I get the following:

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Base64/D7K0n_G9ii0.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Markdown/AREjX_G9ii0.so for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/InteractiveUtils/0TrXF_G9ii0.so for InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/StyledStrings/UcVoM_G9ii0.so for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Unicode/E4Hzs_G9ii0.so for Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/REPL/u0gqU_G9ii0.so for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1185
julia> using Base64
[ Info: Precompiling Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f] 
┌ Debug: Generating object cache file for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:2805
┌ Debug: Loading object cache file /tmp/depot/compiled/v1.11/Base64/D7K0n_j1x0v.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185

julia> exit()

~$ /usr/local/julia-1.11.0-rc2/bin/julia -q --startup-file=no
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Base64/D7K0n_G9ii0.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Markdown/AREjX_G9ii0.so for Markdown [d6f4376e-aef5-505a-96c1-9c027394607a]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/InteractiveUtils/0TrXF_G9ii0.so for InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/StyledStrings/UcVoM_G9ii0.so for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/Unicode/E4Hzs_G9ii0.so for Unicode [4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5]
└ @ Base loading.jl:1185
┌ Debug: Loading object cache file /usr/local/julia-1.11.0-rc2/share/julia/compiled/v1.11/REPL/u0gqU_G9ii0.so for REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb]
└ @ Base loading.jl:1185
julia> using Base64
┌ Debug: Loading object cache file /tmp/depot/compiled/v1.11/Base64/D7K0n_j1x0v.so for Base64 [2a0f44e3-6c83-55bd-87e4-b1978d98bd5f]
└ @ Base loading.jl:1185
matthias314 commented 1 month ago

I can now reproduce it on a different machine, even with a completely empty depot. The problem seems to arise when the path of the depot is the beginning of the path where the Julia distribution is installed. I've taken JULIA_DEPOT_PATH=/tmp/julia and the distribution in /tmp/julia-1.11.0-rc2. If I rename the depot to /tmp/julia-depot, the problem disappears. Renaming the depot to /tmp/juli brings it back, with the error message

julia> using Base64
┌ Debug: Rejecting cache file /tmp/juli/compiled/v1.11/Base64/D7K0n_1PRzk.ji because it is for file @depota-1.11.0-rc2/share/julia/stdlib/v1.11/Base64/src/Base64.jl not file /tmp/julia-1.11.0-rc2/share/julia/stdlib/v1.11/Base64/src/Base64.jl

Note that now @depota-1.11.0-rc2 (with an extra "a") shows up, corresponding to the shortened depot name (from "/tmp/julia" to "/tmp/juli").

fatteneder commented 1 month ago

Note that now @depota-1.11.0-rc2 (with an extra "a") shows up, corresponding to the shortened depot name (from "/tmp/julia" to "/tmp/juli").

I discovered this too recently, but forgot to make a PR right away. Will set up a fix later.