search

JuliaLang / Pkg.jl

Pkg - Package manager for the Julia programming language
https://pkgdocs.julialang.org
Other
609 stars 251 forks source link

[workspace] Only one extension is precompiled #3889

Closed fredrikekre closed 1 month ago

fredrikekre commented 2 months ago

Checkout this branch: https://github.com/JuliaSparse/Metis.jl/tree/fe/precompile-bug

# Remove precompile files
$ rm -rf ~/.julia/compiled/v1.12/Graphs ~/.julia/compiled/v1.12/LightGraphs

# Explicitly instantiate the test env with correct flags (https://github.com/JuliaLang/Pkg.jl/pull/3883/commits/8ab562f72806af458c816dd2f319014f8e05e5cc)
$ julia --project=test --check-bounds=yes -e 'using Pkg; Pkg.precompile()'
Precompiling all packages...
   3286.6 ms  ✓ LightGraphs
   3436.1 ms  ✓ Graphs
   1055.9 ms  ✓ Metis → MetisLightGraphs
    977.8 ms  ✓ SimpleWeightedGraphs
  4 dependencies successfully precompiled in 5 seconds. 51 already precompiled.

$ julia -e 'using Pkg; Pkg.test()'
# Run tests
      Status `~/dev/Metis/test/Project.toml`
⌃ [86223c79] Graphs v1.9.0
  [093fc24a] LightGraphs v1.3.5
  [2679e427] Metis v1.4.1 `.`
  [47aef6b3] SimpleWeightedGraphs v1.4.0
  [9a3f8284] Random v1.11.0
  [2f01184e] SparseArrays v1.11.0
  [8dfed614] Test v1.11.0
      Status `~/dev/Metis/Manifest.toml`
⌅ [ec485272] ArnoldiMethod v0.1.0
  [fa961155] CEnum v0.5.0
  [34da2185] Compat v4.14.0
  [864edb3b] DataStructures v0.18.20
⌃ [86223c79] Graphs v1.9.0
  [d25df0c9] Inflate v0.1.4
  [692b3bcd] JLLWrappers v1.5.0
  [093fc24a] LightGraphs v1.3.5
  [1914dd2f] MacroTools v0.5.13
  [2679e427] Metis v1.4.1 `.`
  [bac558e1] OrderedCollections v1.6.3
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [699a6c99] SimpleTraits v0.9.4
  [47aef6b3] SimpleWeightedGraphs v1.4.0
  [90137ffa] StaticArrays v1.9.3
  [1e83bf80] StaticArraysCore v1.4.2
  [10745b16] Statistics v1.11.1
  [d00139f3] METIS_jll v5.1.2+0
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.11.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.12.0
  [de0858da] Printf v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [1a1011a3] SharedArrays v1.11.0
  [6462fe0b] Sockets v1.11.0
  [2f01184e] SparseArrays v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.1.1+0
  [deac9b47] LibCURL_jll v8.6.0+0
  [e37daf67] LibGit2_jll v1.8.0+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.6+0
  [14a3606d] MozillaCACerts_jll v2024.3.11
  [4536629a] OpenBLAS_jll v0.3.27+1
  [bea87d4a] SuiteSparse_jll v7.6.1+0
  [83775a58] Zlib_jll v1.3.1+0
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.60.0+0
  [3f19e933] p7zip_jll v17.5.0+0
        Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading.
     Testing Running tests...
Precompiling MetisGraphs...
   1111.9 ms  ✓ Metis → MetisGraphs
  1 dependency successfully precompiled in 1 seconds. 53 already precompiled.
Precompiling MetisSimpleWeightedGraphs...
    994.1 ms  ✓ Metis → MetisSimpleWeightedGraphs
  1 dependency successfully precompiled in 1 seconds. 55 already precompiled.
     Testing Metis tests passed

Notice that the MetisGraphs and MetisSimpleWeightedGraphs extensions were missed in the original precompile command and only precompiled when loading the trigger packages in the test script.

KristofferC commented 2 months ago

Simpler repo:

❯ rm -rf ~/.julia/compiled/v1.12/Graphs ~/.julia/compiled/v1.12/LightGraphs  ~/.julia/compiled/v1.12/Metis*

❯ julia --project=test -e 'using Pkg; Pkg.precompile(; timing=true)'                                       
Precompiling all packages...
    716.4 ms  ✓ Metis
   2137.6 ms  ✓ LightGraphs
   2301.4 ms  ✓ Graphs
    894.8 ms  ✓ SimpleWeightedGraphs
   1095.7 ms  ✓ Metis → MetisLightGraphs
  5 dependencies successfully precompiled in 3 seconds. 50 already precompiled.

❯ julia --project=test -e 'using Metis, Graphs'                     
Precompiling MetisGraphs...
  1 dependency successfully precompiled in 1 seconds. 53 already precompiled.
KristofferC commented 1 month ago

I think this is because of the fake dependency that gets introduced in

https://github.com/JuliaLang/julia/blob/88d6633f617665b9cb221694842ed7b8c5e8d951/base/precompilation.jl#L420-L423

which causes

https://github.com/JuliaLang/julia/blob/88d6633f617665b9cb221694842ed7b8c5e8d951/base/precompilation.jl#L546

to fail.

KristofferC commented 1 month ago

And yes, as I expected if the above reasoning was true, https://github.com/JuliaLang/julia/pull/53972 fixes this:

~/JuliaPkgs/Metis.jl fe/precompile-bug
❯ rm -rf ~/.julia/compiled/v1.12/Graphs ~/.julia/compiled/v1.12/LightGraphs  ~/.julia/compiled/v1.12/Metis*                

~/JuliaPkgs/Metis.jl fe/precompile-bug
❯ julia --project=test -e 'include("/home/kc/julia/base/precompilation.jl"); Precompilation.precompilepkgs(; timing=true)' 
Precompiling all packages...
    671.4 ms  ✓ Metis
   1902.9 ms  ✓ LightGraphs
   2112.5 ms  ✓ Graphs
    622.0 ms  ✓ Metis → MetisLightGraphs
    637.8 ms  ✓ Metis → MetisGraphs
    700.0 ms  ✓ SimpleWeightedGraphs
    596.9 ms  ✓ Metis → MetisSimpleWeightedGraphs
  7 dependencies successfully precompiled in 4 seconds. 51 already precompiled.

~/JuliaPkgs/Metis.jl fe/precompile-bug
❯ julia --project=test -e 'using Metis, Graphs'                                                                            

~/JuliaPkgs/Metis.jl fe/precompile-bug
❯
KristofferC commented 1 month ago

Fixed by https://github.com/JuliaLang/julia/pull/53972