The default behavior for addprocs is to initialize workers into the base environment of Julia, rather than the current environment. This is very unintuitive and I think over the course of being a Julia dev I have ran into bugs because of this over ~20 times. Especially for new users, this really feels unintuitive that worker processes are initialized in a different environment, and @everywhere using [MyCustomModule] does not work work out of the box, unless the module is installed in the base env. And sometimes you get really unlucky and end up with the silent behavior of a different version of a package loaded on the workers than the head node.
Therefore I am wondering if it make sense to have default behavior for exeflags set to, e.g., --project=$(import Pkg: project; splitdir(project().path)[1]) or a simpler variation?
I suppose on some clusters without a shared filesystem this would not work, so perhaps there could be a sensible fallback mechanism (but without a shared filesystem, you are bound to encounter issues with incompatible environments anyways). But having something like this as a default feels both safer and more intuitive.
The default behavior for
addprocs
is to initialize workers into the base environment of Julia, rather than the current environment. This is very unintuitive and I think over the course of being a Julia dev I have ran into bugs because of this over ~20 times. Especially for new users, this really feels unintuitive that worker processes are initialized in a different environment, and@everywhere using [MyCustomModule]
does not work work out of the box, unless the module is installed in the base env. And sometimes you get really unlucky and end up with the silent behavior of a different version of a package loaded on the workers than the head node.Therefore I am wondering if it make sense to have default behavior for
exeflags
set to, e.g.,--project=$(import Pkg: project; splitdir(project().path)[1])
or a simpler variation?I suppose on some clusters without a shared filesystem this would not work, so perhaps there could be a sensible fallback mechanism (but without a shared filesystem, you are bound to encounter issues with incompatible environments anyways). But having something like this as a default feels both safer and more intuitive.
This would also make it easier to add processes dynamically internal to a library, rather than having to use hacky solutions like the one I use in SymbolicRegression.jl: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/e07c34681294fdca35a021a0b1a7dc99f334400b/src/Configure.jl#L193-L200
(Also, in a PR fixing this, it would also be useful to add some docs on what is expected for
exeflags
as I had to dive into the code to figure it out)