search

JuliaLang / Pkg.jl

Pkg - Package manager for the Julia programming language
https://pkgdocs.julialang.org
Other
622 stars 263 forks source link

Robustness against corrupt package store #3199

Open maleadt opened 2 years ago

maleadt commented 2 years ago

Recently, PkgEval testing of lots of packages broke because of a corrupt entry in .julia/packages (which is shared and persistent). MWE:

$ rm -rf ~/.julia/packages/Distributions/39PV5
$ mkdir ~/.julia/packages/Distributions/39PV5

julia> using Distributions
ERROR: ArgumentError: Package Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Actual backtrace:


     Testing Bootstrap
      Status `/tmp/jl_oUIXPn/Project.toml`
  [e28b5b4c] Bootstrap v2.3.3
  [a93c6f00] DataFrames v1.3.5
  [31c24e10] Distributions v0.25.70
  [38e38edf] GLM v1.8.0
  [ce6b1742] RDatasets v0.7.7
  [2913bbd2] StatsBase v0.33.21
  [3eaba693] StatsModels v0.6.31
  [9a3f8284] Random `@stdlib/Random`
  [10745b16] Statistics `@stdlib/Statistics`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_oUIXPn/Manifest.toml`
  [e28b5b4c] Bootstrap v2.3.3
  [336ed68f] CSV v0.10.4
  [49dc2e85] Calculus v0.5.1
  [324d7699] CategoricalArrays v0.10.6
  [d360d2e6] ChainRulesCore v1.15.5
  [9e997f8a] ChangesOfVariables v0.1.4
  [944b1d66] CodecZlib v0.7.0
  [34da2185] Compat v4.2.0
  [a8cc5b0e] Crayons v4.1.1
  [9a962f9c] DataAPI v1.10.0
  [a93c6f00] DataFrames v1.3.5
  [864edb3b] DataStructures v0.18.13
  [e2d170a0] DataValueInterfaces v1.0.0
  [b429d917] DensityInterface v0.4.0
  [31c24e10] Distributions v0.25.70
  [ffbed154] DocStringExtensions v0.9.1
  [fa6b7ba4] DualNumbers v0.6.8
  [e2ba6199] ExprTools v0.1.8
  [5789e2e9] FileIO v1.15.0
  [48062228] FilePathsBase v0.9.20
  [1a297f60] FillArrays v0.13.4
  [59287772] Formatting v0.4.2
  [38e38edf] GLM v1.8.0
  [34004b35] HypergeometricFunctions v0.3.11
  [842dd82b] InlineStrings v1.1.4
  [3587e190] InverseFunctions v0.1.7
  [41ab1584] InvertedIndices v1.1.0
  [92d709cd] IrrationalConstants v0.1.1
  [82899510] IteratorInterfaceExtensions v1.0.0
  [692b3bcd] JLLWrappers v1.4.1
  [2ab3a3ac] LogExpFunctions v0.3.18
  [e1d29d7a] Missings v1.0.2
⌃ [78c3b35d] Mocking v0.7.2
  [77ba4419] NaNMath v1.0.1
  [bac558e1] OrderedCollections v1.4.1
  [90014a1f] PDMats v0.11.16
  [69de0a69] Parsers v2.4.0
  [2dfb63ee] PooledArrays v1.4.2
  [21216c6a] Preferences v1.3.0
⌅ [08abe8d2] PrettyTables v1.3.1
  [1fd47b50] QuadGK v2.5.0
  [df47a6cb] RData v0.8.3
  [ce6b1742] RDatasets v0.7.7
  [3cdcf5f2] RecipesBase v1.2.1
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [79098fc4] Rmath v0.7.0
  [6c6a2e73] Scratch v1.1.1
  [91c51154] SentinelArrays v1.3.13
  [1277b4bf] ShiftedArrays v1.0.0
  [a2af1166] SortingAlgorithms v1.0.1
  [276daf66] SpecialFunctions v2.1.7
  [82ae8749] StatsAPI v1.5.0
  [2913bbd2] StatsBase v0.33.21
  [4c63d2b9] StatsFuns v1.0.1
  [3eaba693] StatsModels v0.6.31
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.7.0
  [f269a46b] TimeZones v1.9.0
  [3bb67fe8] TranscodingStreams v0.9.9
  [ea10d353] WeakRefStrings v1.4.2
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.3.0+0
  [0dad84c5] ArgTools v1.1.1 `@stdlib/ArgTools`
  [56f22d72] Artifacts `@stdlib/Artifacts`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [ade2ca70] Dates `@stdlib/Dates`
  [f43a241f] Downloads v1.6.0 `@stdlib/Downloads`
  [7b1f6079] FileWatching `@stdlib/FileWatching`
  [9fa8497b] Future `@stdlib/Future`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [4af54fe1] LazyArtifacts `@stdlib/LazyArtifacts`
  [b27032c2] LibCURL v0.6.3 `@stdlib/LibCURL`
  [76f85450] LibGit2 `@stdlib/LibGit2`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [a63ad114] Mmap `@stdlib/Mmap`
  [ca575930] NetworkOptions v1.2.0 `@stdlib/NetworkOptions`
  [44cfe95a] Pkg v1.8.0 `@stdlib/Pkg`
  [de0858da] Printf `@stdlib/Printf`
  [3fa0cd96] REPL `@stdlib/REPL`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [6462fe0b] Sockets `@stdlib/Sockets`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [10745b16] Statistics `@stdlib/Statistics`
  [4607b0f0] SuiteSparse `@stdlib/SuiteSparse`
  [fa267f1f] TOML v1.0.0 `@stdlib/TOML`
  [a4e569a6] Tar v1.10.0 `@stdlib/Tar`
  [8dfed614] Test `@stdlib/Test`
  [cf7118a7] UUIDs `@stdlib/UUIDs`
  [4ec0a83e] Unicode `@stdlib/Unicode`
  [e66e0078] CompilerSupportLibraries_jll v0.5.2+0 `@stdlib/CompilerSupportLibraries_jll`
  [deac9b47] LibCURL_jll v7.84.0+0 `@stdlib/LibCURL_jll`
  [29816b5a] LibSSH2_jll v1.10.2+0 `@stdlib/LibSSH2_jll`
  [c8ffd9c3] MbedTLS_jll v2.28.0+0 `@stdlib/MbedTLS_jll`
  [14a3606d] MozillaCACerts_jll v2022.2.1 `@stdlib/MozillaCACerts_jll`
  [4536629a] OpenBLAS_jll v0.3.21+0 `@stdlib/OpenBLAS_jll`
  [05823500] OpenLibm_jll v0.8.1+0 `@stdlib/OpenLibm_jll`
  [bea87d4a] SuiteSparse_jll v5.10.1+0 `@stdlib/SuiteSparse_jll`
  [83775a58] Zlib_jll v1.2.12+3 `@stdlib/Zlib_jll`
  [8e850b90] libblastrampoline_jll v5.1.1+0 `@stdlib/libblastrampoline_jll`
  [8e850ede] nghttp2_jll v1.48.0+0 `@stdlib/nghttp2_jll`
  [3f19e933] p7zip_jll v17.4.0+0 `@stdlib/p7zip_jll`
        Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading.
     Testing Running tests...
ERROR: LoadError: ArgumentError: Package Distributions [31c24e10-a181-5473-b8eb-7969acd0382f] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1390
 [2] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1281
 [3] macro expansion
   @ ./loading.jl:1261 [inlined]
 [4] macro expansion
   @ ./lock.jl:267 [inlined]
 [5] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1224
 [6] include
   @ ./Base.jl:434 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1638
 [8] top-level scope
   @ stdin:1
in expression starting at /home/pkgeval/.julia/packages/Bootstrap/6veet/src/Bootstrap.jl:1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Bootstrap [e28b5b4c-05e8-5b66-bc03-6f0c0a0a06e0] to /home/pkgeval/.julia/compiled/v1.9/Bootstrap/jl_ZDIgbE.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1789
  [3] compilecache
    @ ./loading.jl:1733 [inlined]
  [4] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1421
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1281
  [6] macro expansion
    @ ./loading.jl:1261 [inlined]
  [7] macro expansion
    @ ./lock.jl:267 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1224
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [10] top-level scope
    @ ~/.julia/packages/Bootstrap/6veet/test/runtests.jl:8
 [11] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [12] top-level scope
    @ none:6
in expression starting at /home/pkgeval/.julia/packages/Bootstrap/6veet/test/test-non-parametric.jl:1
in expression starting at /home/pkgeval/.julia/packages/Bootstrap/6veet/test/runtests.jl:6

It would be great if Julia/Pkg were robust against such corruption, reinstalling the affected package when needed. Alternatively, avoiding the corruption in the first place would be great as well. I think it was either caused by multiple testers writing to the package store at the same time (which would require locking), or a well-timed cancellation signal leaving the folder in a bad state (which would require the final operation to be atomic). I'm not familiar with Pkg so don't know if either of these is already the case.

fredrikekre commented 2 years ago

Related: #1683, #1717.

rfourquet commented 1 year ago

Another case where corruption happens is when the disk becomes full while downloading a package.