Feature request: Support instantiation of multiple julia environments in parallel

[schlichtanders]

With Julia 1.9 it gets more important to do proper precompilation ahead of time. I am using Pluto and like to instantiate all notebook environments with high speed.

Currently, when just spawning multiple julia processes and instantiating several environments, I get a GitError as soon as two environments have the same dependency (there is probably some randomness involved, but it failes constantly for me - there are several overlaps)

GitError(Code:ERROR, Class:OS, could not open '/home/myhome/.julia/packages/MyPackage/1opWe/Project.toml' for writing: Permission denied)

It would be really great to have a performant way of instantiating multiple environments at once.

EDIT: Just realized, that I only tested this Julia 1.8.5, I am going to test on Julia 1.9 immediately Applies to 1.9 as well

[vchuravy]

What's the full backtrace for the GitError?

[schlichtanders]

Meanwhile I upgraded the Manifests (before they still were 1.8.5 manifests).

So far rerunning again produced an Error with the above mentioned MyPackage, however the error was not printed to the console. I will try rerunning and hope I get the stacktrace as well.

In addition the following error occured:

IOError: readdir("/home/jolin_user/.julia/packages/ColorSchemes/lTeyu/data"): no such file or directory (ENOENT)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:920
  [2] wait()
    @ Base ./task.jl:984
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] take_unbuffered(c::Channel{Tuple{String, Vector{String}, Vector{String}}})
    @ Base ./channels.jl:473
  [6] take!
    @ ./channels.jl:450 [inlined]
  [7] iterate(c::Channel{Tuple{String, Vector{String}, Vector{String}}}, state::Nothing)
    @ Base ./channels.jl:592
  [8] set_readonly(path::String)
    @ Pkg /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/utils.jl:54
  [9] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:854 [inlined]
 [10] (::Pkg.Operations.var"#47#52"{Bool, Pkg.Types.Context, Tuple{SubString{String}, Dict{Base.UUID, Base.SHA1}}, Channel{Any}, Channel{NamedTuple{(:pkg, :urls, :path), Tuple{Pkg.Types.PackageEntry, Set{String}, String}}}})()
    @ Pkg.Operations ./task.jl:514
Stacktrace:
  [1] pkgerror(::String, ::Vararg{String})
    @ Pkg.Types /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
  [2] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:874 [inlined]
  [3] macro expansion
    @ ./task.jl:476 [inlined]
  [4] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:815
  [5] download_source
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
  [6] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1738
  [7] instantiate
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1640 [inlined]
  [8] instantiate(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
  [9] instantiate()
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
 [10] top-level scope

Everything is hard to reproduce - running my setup again sometimes does not show any error at all.

[schlichtanders]

another stacktrace

Error when installing package DataFrames:
IOError: readdir("/home/jolin_user/.julia/packages/DataFrames/LteEl/docs/src"): no such file or directory (ENOENT)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:920
  [2] wait()
    @ Base ./task.jl:984
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] take_unbuffered(c::Channel{Tuple{String, Vector{String}, Vector{String}}})
    @ Base ./channels.jl:473
  [6] take!
    @ ./channels.jl:450 [inlined]
  [7] iterate(c::Channel{Tuple{String, Vector{String}, Vector{String}}}, state::Nothing)
    @ Base ./channels.jl:592
  [8] set_readonly(path::String)
    @ Pkg /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/utils.jl:54
  [9] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:854 [inlined]
 [10] (::Pkg.Operations.var"#47#52"{Bool, Pkg.Types.Context, Tuple{SubString{String}, Dict{Base.UUID, Base.SHA1}}, Channel{Any}, Channel{NamedTuple{(:pkg, :urls, :path), Tuple{Pkg.Types.PackageEntry, Set{String}, String}}}})()
    @ Pkg.Operations ./task.jl:514
Stacktrace:
  [1] pkgerror(::String, ::Vararg{String})
    @ Pkg.Types /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
  [2] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:874 [inlined]
  [3] macro expansion
    @ ./task.jl:476 [inlined]
  [4] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:815
  [5] download_source
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
  [6] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1738
  [7] instantiate
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1640 [inlined]
  [8] instantiate(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
  [9] instantiate()
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
 [10] top-level scope

unfortunately, the original GitError I could not reproduce so far (maybe it was depending on the old manifest)

[schlichtanders]

I now reverted my setup to use the old manifest and was able to replicate the GitError

ERROR: GitError(Code:ERROR, Class:OS, could not open '/home/jolin_user/.julia/packages/MyPackage/1opWe/Project.toml' for writing: Permission denied)
Stacktrace:
  [1] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/error.jl:111 [inlined]
  [2] checkout_tree(repo::LibGit2.GitRepo, obj::LibGit2.GitTree; options::LibGit2.CheckoutOptions)
    @ LibGit2 /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:358
  [3] checkout_tree
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:354 [inlined]
  [4] checkout_tree_to_path(repo::LibGit2.GitRepo, tree::LibGit2.GitTree, path::String)
    @ Pkg.GitTools /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/GitTools.jl:84
  [5] install_git(io::Base.TTY, uuid::Base.UUID, name::String, hash::Base.SHA1, urls::Set{String}, version_path::String)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:699
  [6] download_source(ctx::Pkg.Types.Context; readonly::Bool)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:905
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:791 [inlined]
  [8] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1738
  [9] instantiate
    @ /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1640 [inlined]
 [10] instantiate(; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
 [11] instantiate()
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639
   Installed libaom_jll ─────────────────── v3.4.0+0
  [1] macro expansion
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/error.jl:111 [inlined]
  [2] checkout_tree(repo::LibGit2.GitRepo, obj::LibGit2.GitTree; options::LibGit2.CheckoutOptions)
    @ LibGit2 /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:358
  [3] checkout_tree
    @ /usr/local/julia/share/julia/stdlib/v1.9/LibGit2/src/repository.jl:354 [inlined]
  [4] checkout_tree_to_path(repo::LibGit2.GitRepo, tree::LibGit2.GitTree, path::String)
    @ Pkg.GitTools /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/GitTools.jl:84
  [5] install_git(io::Base.TTY, uuid::Base.UUID, name::String, hash::Base.SHA1, urls::Set{String}, version_path::String)
    @ Pkg.Operations /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:699
    @ Pkg.API /usr/local/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:1639

[vchuravy]

So some of these things make sense. The checkouts of the packages are read-only.

So GitError(Code:ERROR, Class:OS, could not open '/home/jolin_user/.julia/packages/MyPackage/1opWe/Project.toml' for writing: Permission denied) is sensible, but the real question is: Who is calling instantiate on that copy?

What operations are you doing? How could one reproduce these errors?

[schlichtanders]

I have a couple of directories with Project.toml and Manifest.toml (actually there is really only these two files, which have been extracted from Pluto notebooks and written to their separate directories)

then I am doing just

@sync for environment_dir in environment_directories
    @async run(`julia --project $environment_dir -e 'import Pkg; Pkg.instantiate()'`)
end

I am sorry, but creating the environment_directories is a bit tough right now, I hope that the error stacks above are enough to spot which additions would be needed to make several Pkg.instantiate() safely work in parallel.

[vchuravy]

I am sorry but what this points to is that how ever you generate environment_directories is buggy.

You simply can't call instantiate on .julia/packages/** those directories are marked intentionally read-only.

So the error you are seeing is correct, trying to write to that directory (either to modify Project or to create Manifest) is not allowed

[schlichtanders]

I bet that the environment_directories are fully fine - they were generated with Pluto.jl automatically. And everything works without problems when done iteratively.

MyPackage is a bit special in the sense that it comes from a LocalRegistry and not general.

[vchuravy]

Without a MWE it won't be possible to determine why you are seeing this error. As a I said the last error in particular hints at someone calling instantiate on a .julia/packages/** directory. That is not supported, these are not environments, but local immutable copies of the package.

The only thing you should see races on are cache files in .julia/compiled but generally Julia supports parallel compilation (modulo the fact that in case of a race the last attempt wins).

[schlichtanders]

Okay, here a reproducible example. It is not minimal (I gave it an attempt to minifiy it further, but without success.), but at least it is reproducible. Concretely it mirrors my own experiments above.

1. start docker with docker run -it --rm julia:1.9 bash
2. execute the following

cd root
apt-get update && apt-get install -y git
git clone https://github.com/jolin-io/JolinWorkspaceTemplate/
cd JolinWorkspaceTemplate

# This will lead to the GitError
git reset --hard 3266327

# This will lead to the other Errors
# git reset --hard cd3614b

# julia
julia -e '
    import Pkg
    Pkg.Registry.add("General")
    Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/jolin-io/JolinRegistry.jl"))
    Pkg.add("JolinPlutoCICD")

    using JolinPlutoCICD
    workflows = JolinPlutoCICD.get_all_workflow_paths("./workflows")
    envs = Dict(w => JolinPlutoCICD.create_pluto_env(w) for w in workflows)

    @sync for (_, envdir) in envs
        @show envdir
        @async Base.run(`julia --project=$envdir -e "import Pkg; Pkg.instantiate()"`)
    end
'

please mind that there are two different git reset --hard which will point to two different tests.

[vchuravy]

Thank you for the MWE it helped me understand what you want to achieve.

Julia itself doesn't instantiate, it defers that to Pkg.jl

I suspect that Pkg would need to either implement PIDLocks (similar to in https://github.com/JuliaLang/julia/pull/49052) or try use atomic move (fraught with pitfalls) to make sure that other Julia processes don't see "partially instantiated" or race upon the ".julia/packages/..." directories.

[schlichtanders]

@IanButterworth would this also be solved by the mentioned issue

[charleskawczynski]

I'm seeing this issue too, almost reproducibly so, in buildkite CI for https://github.com/CliMA/ClimaComms.jl

[charleskawczynski]

Is this a dup of #1219?

[vchuravy]

Not quite. Julia 1.10 will be better around races on cache-files, but there is still an open question around two separate processes calling instantiate concurrently, on the same project or a project that contains the same dependency.