JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.9k stars 5.49k forks source link

`addprocs` should activate workers in current project #49003

Closed MilesCranmer closed 1 year ago

MilesCranmer commented 1 year ago

The default behavior for addprocs is to initialize workers into the base environment of Julia, rather than the current environment. This is very unintuitive and I think over the course of being a Julia dev I have ran into bugs because of this over ~20 times. Especially for new users, this really feels unintuitive that worker processes are initialized in a different environment, and @everywhere using [MyCustomModule] does not work work out of the box, unless the module is installed in the base env. And sometimes you get really unlucky and end up with the silent behavior of a different version of a package loaded on the workers than the head node.

Therefore I am wondering if it make sense to have default behavior for exeflags set to, e.g., --project=$(import Pkg: project; splitdir(project().path)[1]) or a simpler variation?

I suppose on some clusters without a shared filesystem this would not work, so perhaps there could be a sensible fallback mechanism (but without a shared filesystem, you are bound to encounter issues with incompatible environments anyways). But having something like this as a default feels both safer and more intuitive.

This would also make it easier to add processes dynamically internal to a library, rather than having to use hacky solutions like the one I use in SymbolicRegression.jl: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/e07c34681294fdca35a021a0b1a7dc99f334400b/src/Configure.jl#L193-L200

    @everywhere procs begin
        Base.MainInclude.eval(
            quote
                using Pkg
                Pkg.activate($$project_path)
            end,
        )
    end

(Also, in a PR fixing this, it would also be useful to add some docs on what is expected for exeflags as I had to dive into the code to figure it out)

giordano commented 1 year ago

Duplicate of the already-fixed #28781?

MilesCranmer commented 1 year ago

Awesome. What Julia version will it be in?

giordano commented 1 year ago

1.9