search

JuliaLang / Pkg.jl

Pkg - Package manager for the Julia programming language
https://pkgdocs.julialang.org
Other
621 stars 269 forks source link

Precompile after `Pkg.add` tries to precompile dependencies that were removed #3863

Closed Keno closed 6 months ago

Keno commented 7 months ago

In the following, the precompile after the second add should not have attempted to precompile Enzyme, because it got removed from the manifest. Indeed, and explicit precompile after the add succeeds. It looks like the precompile after the add may be issued with the wrong (outdated) list of packages.

(test) pkg> add SciMLSensitivity
   Resolving package versions...
    Updating `~/test/Project.toml`
  [1ed8b502] + SciMLSensitivity v7.56.2
    Updating `~/test/Manifest.toml`
  [47edcb42] + ADTypes v0.2.7
  [621f4979] + AbstractFFTs v1.5.0
  [7d9f7c33] + Accessors v0.1.36
  [79e6a3ab] + Adapt v4.0.4
⌅ [ec485272] + ArnoldiMethod v0.2.0
  [4fba245c] + ArrayInterface v7.9.0
  [4c555306] + ArrayLayouts v1.8.0
  [a9b6321e] + Atomix v0.1.0
  [62783981] + BitTwiddlingConvenienceFunctions v0.1.5
  [fa961155] + CEnum v0.5.0
  [2a0fbf3d] + CPUSummary v0.2.4
  [49dc2e85] + Calculus v0.5.1
  [7057c7e9] + Cassette v0.3.13
  [082447d4] + ChainRules v1.63.0
  [d360d2e6] + ChainRulesCore v1.23.0
  [fb6a15b2] + CloseOpenIntervals v0.1.12
  [38540f10] + CommonSolve v0.2.4
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v4.14.0
  [a33af91c] + CompositionsBase v0.1.2
  [2569d6c7] + ConcreteStructs v0.2.3
  [187b0558] + ConstructionBase v1.5.5
  [adafc99b] + CpuId v0.3.1
  [9a962f9c] + DataAPI v1.16.0
  [864edb3b] + DataStructures v0.18.18
  [e2d170a0] + DataValueInterfaces v1.0.0
  [2b5f629d] + DiffEqBase v6.149.0
  [459566f4] + DiffEqCallbacks v3.5.0
  [77a26b50] + DiffEqNoiseProcess v5.21.0
  [163ba53b] + DiffResults v1.1.0
  [b552c78f] + DiffRules v1.15.1
  [b4f34e82] + Distances v0.10.11
  [31c24e10] + Distributions v0.25.107
  [ffbed154] + DocStringExtensions v0.9.3
  [fa6b7ba4] + DualNumbers v0.6.8
  [da5c29d0] + EllipsisNotation v1.8.0
  [4e289a0a] + EnumX v1.0.4
  [7da242da] + Enzyme v0.11.20
⌅ [f151be2c] + EnzymeCore v0.6.6
  [d4d017d3] + ExponentialUtilities v1.26.1
  [e2ba6199] + ExprTools v0.1.10
  [7034ab61] + FastBroadcast v0.2.8
  [9aa1b823] + FastClosures v0.3.2
  [29a986be] + FastLapackInterface v2.0.2
  [1a297f60] + FillArrays v1.10.0
  [6a86dc24] + FiniteDiff v2.23.0
  [f6369f11] + ForwardDiff v0.10.36
  [f62d2435] + FunctionProperties v0.1.2
  [069b7b12] + FunctionWrappers v1.1.3
  [77dc65aa] + FunctionWrappersWrappers v0.1.3
  [d9f16b24] + Functors v0.4.9
  [0c68f7d7] + GPUArrays v10.0.2
  [46192b85] + GPUArraysCore v0.1.6
⌅ [61eb1bfa] + GPUCompiler v0.25.0
  [c145ed77] + GenericSchur v0.5.4
  [86223c79] + Graphs v1.9.0
  [3e5b6fbb] + HostCPUFeatures v0.1.16
  [34004b35] + HypergeometricFunctions v0.3.23
  [7869d1d1] + IRTools v0.4.12
  [615f187c] + IfElse v0.1.1
  [d25df0c9] + Inflate v0.1.4
  [3587e190] + InverseFunctions v0.1.13
  [92d709cd] + IrrationalConstants v0.2.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [ccbc3e58] + JumpProcesses v9.11.1
  [ef3ab10e] + KLU v0.6.0
  [63c18a36] + KernelAbstractions v0.9.18
  [ba0b0d4f] + Krylov v0.9.5
  [929cbde3] + LLVM v6.6.2
  [10f19ff3] + LayoutPointers v0.1.15
  [5078a376] + LazyArrays v1.9.0
  [2d8b4e74] + LevyArea v1.0.0
  [d3d80556] + LineSearches v7.2.0
  [7ed4a6bd] + LinearSolve v2.28.0
  [2ab3a3ac] + LogExpFunctions v0.3.27
  [bdcacae8] + LoopVectorization v0.12.169
  [1914dd2f] + MacroTools v0.5.13
  [d125e4d3] + ManualMemory v0.1.8
  [a3b82374] + MatrixFactorizations v2.1.0
  [bb5d69b7] + MaybeInplace v0.1.2
  [e1d29d7a] + Missings v1.1.0
  [46d2c3a1] + MuladdMacro v0.2.4
  [d41bc354] + NLSolversBase v7.8.3
  [2774e3e8] + NLsolve v4.5.1
  [872c559c] + NNlib v0.9.13
  [77ba4419] + NaNMath v1.0.2
  [8913a72c] + NonlinearSolve v3.9.1
  [d8793406] + ObjectFile v0.4.1
  [6fe1bfb0] + OffsetArrays v1.13.0
  [429524aa] + Optim v1.9.4
  [3bd65402] + Optimisers v0.3.2
  [bac558e1] + OrderedCollections v1.6.3
  [1dea7af3] + OrdinaryDiffEq v6.74.1
  [90014a1f] + PDMats v0.11.31
  [65ce6f38] + PackageExtensionCompat v1.0.2
  [d96e819e] + Parameters v0.12.3
  [e409e4f3] + PoissonRandom v0.4.4
  [f517fe37] + Polyester v0.7.12
  [1d0040c9] + PolyesterWeave v0.2.1
  [85a6dd25] + PositiveFactorizations v0.2.4
  [d236fae5] + PreallocationTools v0.4.20
  [aea7be01] + PrecompileTools v1.2.1
  [21216c6a] + Preferences v1.4.3
  [1fd47b50] + QuadGK v2.9.4
  [74087812] + Random123 v1.7.0
  [e6cf234a] + RandomNumbers v1.5.3
  [c1ae055f] + RealDot v0.1.0
  [3cdcf5f2] + RecipesBase v1.3.4
  [731186ca] + RecursiveArrayTools v3.13.0
  [f2c3362d] + RecursiveFactorization v0.2.21
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [ae5879a3] + ResettableStacks v1.1.1
  [37e2e3b7] + ReverseDiff v1.15.1
  [79098fc4] + Rmath v0.7.1
  [7e49a35a] + RuntimeGeneratedFunctions v0.5.12
  [94e857df] + SIMDTypes v0.1.0
  [476501e8] + SLEEFPirates v0.6.42
  [0bca4576] + SciMLBase v2.31.0
  [c0aeaf25] + SciMLOperators v0.3.8
  [1ed8b502] + SciMLSensitivity v7.56.2
  [53ae85a6] + SciMLStructures v1.1.0
  [6c6a2e73] + Scratch v1.2.1
  [efcf1570] + Setfield v1.1.1
  [727e6d20] + SimpleNonlinearSolve v1.7.0
  [699a6c99] + SimpleTraits v0.9.4
  [ce78b400] + SimpleUnPack v1.1.0
  [a2af1166] + SortingAlgorithms v1.2.1
  [47a9eef4] + SparseDiffTools v2.17.0
  [dc90abb0] + SparseInverseSubset v0.1.2
  [e56a9233] + Sparspak v0.3.9
  [276daf66] + SpecialFunctions v2.3.1
  [aedffcd0] + Static v0.8.10
  [0d7ed370] + StaticArrayInterface v1.5.0
  [90137ffa] + StaticArrays v1.9.3
  [1e83bf80] + StaticArraysCore v1.4.2
  [10745b16] + Statistics v1.11.1
  [82ae8749] + StatsAPI v1.7.0
  [2913bbd2] + StatsBase v0.34.3
  [4c63d2b9] + StatsFuns v1.3.1
  [789caeaf] + StochasticDiffEq v6.65.1
  [7792a7ef] + StrideArraysCore v0.5.2
  [09ab397b] + StructArrays v0.6.18
  [53d494c1] + StructIO v0.3.0
  [2efcf032] + SymbolicIndexingInterface v0.3.15
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.11.1
  [8290d209] + ThreadingUtilities v0.5.2
  [a759f4b9] + TimerOutputs v0.5.23
  [9f7883ad] + Tracker v0.2.33
  [d5829a12] + TriangularSolve v0.1.21
  [410a4b4d] + Tricks v0.1.8
  [781d530d] + TruncatedStacktraces v1.4.0
  [3a884ed6] + UnPack v1.0.2
  [013be700] + UnsafeAtomics v0.2.1
  [d80eeb9a] + UnsafeAtomicsLLVM v0.1.3
  [3d5dd08c] + VectorizationBase v0.21.65
  [19fa3120] + VertexSafeGraphs v0.2.0
  [e88e6eb3] + Zygote v0.6.69
  [700de1a5] + ZygoteRules v0.2.5
⌅ [7cc45869] + Enzyme_jll v0.0.102+0
  [1d5cc7b8] + IntelOpenMP_jll v2024.0.2+0
  [dad2f222] + LLVMExtra_jll v0.0.29+0
  [856f044c] + MKL_jll v2024.0.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [f50d1b31] + Rmath_jll v0.4.0+0
  [0dad84c5] + ArgTools v1.1.2
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [8ba89e20] + Distributed v1.11.0
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching v1.11.0
  [9fa8497b] + Future v1.11.0
  [b77e0a4c] + InteractiveUtils v1.11.0
  [4af54fe1] + LazyArtifacts v1.11.0
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2 v1.11.0
  [8f399da3] + Libdl v1.11.0
  [37e2e46d] + LinearAlgebra v1.11.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [a63ad114] + Mmap v1.11.0
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.12.0
  [de0858da] + Printf v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization v1.11.0
  [1a1011a3] + SharedArrays v1.11.0
  [6462fe0b] + Sockets v1.11.0
  [2f01184e] + SparseArrays v1.11.0
  [f489334b] + StyledStrings v1.11.0
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test v1.11.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [e66e0078] + CompilerSupportLibraries_jll v1.1.1+0
  [deac9b47] + LibCURL_jll v8.6.0+0
  [e37daf67] + LibGit2_jll v1.8.0+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.6+0
  [14a3606d] + MozillaCACerts_jll v2024.3.11
  [4536629a] + OpenBLAS_jll v0.3.26+2
  [05823500] + OpenLibm_jll v0.8.1+2
  [bea87d4a] + SuiteSparse_jll v7.6.1+0
  [83775a58] + Zlib_jll v1.3.1+0
  [8e850b90] + libblastrampoline_jll v5.8.0+1
  [8e850ede] + nghttp2_jll v1.60.0+0
  [3f19e933] + p7zip_jll v17.5.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Precompiling all packages...
  ✗ Cassette
  ✗ IRTools
  ✗ Enzyme
  ✗ Zygote
  ✗ FunctionProperties
  ✗ Zygote → ZygoteTrackerExt
  ✗ Zygote → ZygoteDistancesExt
  ✗ RecursiveArrayTools → RecursiveArrayToolsReverseDiffExt
  ✗ SparseDiffTools → SparseDiffToolsZygoteExt
  ✗ RecursiveArrayTools → RecursiveArrayToolsZygoteExt
  ✗ SciMLBase → SciMLBaseZygoteExt
  ✗ Enzyme → EnzymeSpecialFunctionsExt
  ✗ SparseDiffTools → SparseDiffToolsEnzymeExt
  ✗ LinearSolve → LinearSolveEnzymeExt
  ✗ DiffEqBase → DiffEqBaseEnzymeExt
  ✗ SimpleNonlinearSolve → SimpleNonlinearSolveZygoteExt
  ✗ NonlinearSolve → NonlinearSolveZygoteExt

(test) pkg> add "https://github.com/Keno/SciMLSensitivity.jl"#kf/mindep
    Updating git-repo `https://github.com/Keno/SciMLSensitivity.jl`
   Resolving package versions...
    Updating `~/test/Project.toml`
  [1ed8b502] ~ SciMLSensitivity v7.56.2 ⇒ v7.56.2 `https://github.com/Keno/SciMLSensitivity.jl#kf/mindep`
    Updating `~/test/Manifest.toml`
  [621f4979] - AbstractFFTs v1.5.0
  [7057c7e9] - Cassette v0.3.13
  [082447d4] - ChainRules v1.63.0
  [7da242da] - Enzyme v0.11.20
  [f62d2435] - FunctionProperties v0.1.2
  [0c68f7d7] - GPUArrays v10.0.2
  [61eb1bfa] - GPUCompiler v0.25.0
  [7869d1d1] - IRTools v0.4.12
  [d8793406] - ObjectFile v0.4.1
  [c1ae055f] - RealDot v0.1.0
  [37e2e3b7] - ReverseDiff v1.15.1
  [1ed8b502] ~ SciMLSensitivity v7.56.2 ⇒ v7.56.2 `https://github.com/Keno/SciMLSensitivity.jl#kf/mindep`
  [6c6a2e73] - Scratch v1.2.1
  [dc90abb0] - SparseInverseSubset v0.1.2
  [09ab397b] - StructArrays v0.6.18
  [53d494c1] - StructIO v0.3.0
  [e88e6eb3] - Zygote v0.6.69
  [700de1a5] - ZygoteRules v0.2.5
  [7cc45869] - Enzyme_jll v0.0.102+0
Precompiling all packages...
  Progress [==========>                              ]  8/35
  ✗ IRTools
  ✗ Enzyme
  ✗ Zygote
  ✗ Zygote → ZygoteTrackerExt
  ✗ RecursiveArrayTools → RecursiveArrayToolsReverseDiffExt
  ✗ SparseDiffTools → SparseDiffToolsZygoteExt
  ✗ RecursiveArrayTools → RecursiveArrayToolsZygoteExt
  ✗ SciMLBase → SciMLBaseZygoteExt
KristofferC commented 7 months ago

This is kind of strange because the Base precompilation parses the TOML files and the precompilation happens after the TOML files are updated. So it is unclear to me how these packages gets found at all by Base package precompilation. Will investigate.

KristofferC commented 7 months ago

It seems the manifest is not pruned properly. After running the operations above there is still a deps.Enzyme in the Manifest.toml even though nothing is referring to it.

Enzyme gets added as a dep to DiffEqBase here:

https://github.com/JuliaLang/Pkg.jl/blob/8cc835c7d455076f92c5d63a960b0ca0dfd7b2ce/src/Operations.jl#L218

Must have something to do with that Enzyme being a weak dep. So deps_map is wrong?

KristofferC commented 7 months ago

The manifest pruning issue seems to be a problem all the way back to 1.10... But even so, this still seems to indicate a regression in the Base precompilation code because I don't understand why it would ever "reach" the orphan Enzyme.

Attaching the problematic env here so I can easily look into that from home.

env.zip

IanButterworth commented 7 months ago

Related? https://github.com/JuliaLang/Pkg.jl/pull/3720

KristofferC commented 7 months ago

That only add the extension data so I don't see how that would do anything related to this.

KristofferC commented 6 months ago

This should not happen anymore after https://github.com/JuliaLang/Pkg.jl/pull/3864