install into a single global directory?

[stevengj]

Currently we install into the deps directory. However, in Julia 1.0, packages are no longer updated in place, this means that every time you update Conda.jl it needs to re-install anaconda, which is wasteful and breaks things like PyCall that expect libpython to stay in one place.

My thinking is that we should put the root environment into ~/.julia/$JLENV/conda$(CONDA_JL_VERSION) instead, where JLENV is the current Julia package environment. This way, we will only have a single Anaconda installation (per Python major version) that persists across updates.

However, I must admit that I don't fully grok how Pkg environments work. From reading the docs, I guess there is a stack of environments. @StefanKarpinski, is there a way to tell which environment Conda.jl was installed into? Am I thinking about this in the right way?

[tkf]

Isn't it wasteful to have the base conda environment for each Julia environment? I'd suggest:

• ~/.julia/data/Conda/base: base conda installation used from all versions of Julia and Conda.jl
• ~/.julia/data/Conda/envs/v$X.$Y: default conda environment to be used for Julia version $X.$Y.
• ~/.julia/data/Conda/envs/$NAMED_ENV: conda environment to be used for named environment $NAMED_ENV (i.e., the idea is to map ~/.julia/environments/$DIRECTORY to ~/.julia/data/Conda/envs/$DIRECTORY, including ~/.julia/environments/v$X.$Y).
• $PROJECT_PATH/.condajl (where $PROJECT_PATH is the directory s.t. $PROJECT_PATH/Project.toml exists): conda environment to be used for the project/environment at $PROJECT_PATH. Ideally this should be optional since creating a conda environment is much more expensive than how Pkg3 creates a Julia environment. But I think it requires package options https://github.com/JuliaLang/Juleps/issues/38

Notes: The reason why above scheme (i.e., sharing ~/.julia/data/Conda/base across all Julia versions and environments) is useful is that conda has some basic de-duplication mechanisms. For example, it tries to use hard-link in the conda environments when it can. The conda installation directory (~/.julia/data/Conda/base above; usually ~/miniconda3 etc.) has some global cache such as pkgs containing downloaded package archives and decompressed directories of them. Using such mechanism to save disk space requires to share the base installation.

Some discussion points:

• In the above suggestion I am assuming that ~/.julia/data/$PACKAGE is the location that is dedicated to $PACKAGE. However, this does not fit well with Pkg3's model where a package is identified by a UUID. So maybe it should be something like ~/.julia/data/8f4d0f93-b110-5947-807f-2305c1781a2d?
• Maybe use the standard location ~/miniconda instead of ~/.julia/data/Conda/base?
• Is data the right name? opt? share?

Edit: swap "data" and "share" Edit2: Add "Notes"

[stevengj]

I agree it's fine for different Pkg environments to correspond to different conda virtualenvs.

share in Unix refers to a directory for architecture-independent data. I think ~/.julia/conda{2,3} is fine here and will be convenient — the Conda.jl package is kind of in a special situation because it would be unwise for other packages to install whole Anaconda distros.

I don't think we should use ~/miniconda — the whole point of Conda.jl is to install a Julia-specific Anaconda distro so that it doesn't get messed up by whatever other stuff the user might have installed.

[tkf]

I thought it'd be nice to have a scheme for package-specific data directory. For example, dataset library (like RDatasets.jl) could use such location across all environments. But this is more like Pkg.jl enhancement idea.

I understand that that the aim of Conda.jl is to isolate it from user's ~/{mini,ana}conda. My point was that you only need to have a single conda installation for Julia-specific usage. Everything else can be conda's (virtual) environment. Those environments are isolated from the base environment (in principle, unless conda have some critical bugs). You can even create Python 2 conda environment with Miniconda3. Furthermore, you can use ~/miniconda to "bootstrap" Conda.jl's main environment by installing conda package in it. This way, you don't need to touch ~/miniconda base environment. But this last point was probably too eager and was not the main point.

[stevengj]

I'm just worried about user-maintained ~/miniconda base environments being broken in some way — I've seen too many bit-rotted Python installations to trust something we find in a non-Julia directory.

[stevengj]

(See https://github.com/oxinabox/DataDeps.jl for other kinds of data.)

[tkf]

I'm just worried about user-maintained ~/miniconda base environments being broken in some way

Sure, I understand the worry. How about reusing the same miniconda installation for all Julia versions and all Julia environments? (They of course can have different environments.)

DataDeps.jl

It looks like they have a similar discussion too: https://github.com/oxinabox/DataDeps.jl/issues/48

[stevengj]

The lack of persistent package options (JuliaLang/Juleps#38) is a problem here too, because we currently have no way of "remembering" whether the user selected Python 2 or Python 3 or some custom environment.

It's pretty urgent that we get some fix here, even if it is suboptimal. Upgrading Conda.jl currently takes forever, breaks PyCall (because the libpython path changes), and wastes gigabytes of space.

[tkf]

Why not use ~/.julia/data/Conda/envs/v$(VERSION.major).$(VERSION.minor)? This would be forward-compatible to what I suggested in https://github.com/JuliaLang/Pkg.jl/issues/777#issuecomment-428058336. Or maybe even ~/.julia/environments/v$(VERSION.major).$(VERSION.minor)/condajl?

Isn't it simple to fix once the location is decided?

[StefanKarpinski]

The Julia version doesn’t seem necessary or sufficient for isolating Conda setups.

[tkf]

I agree. But I don't think it's reasonable to install miniconda for each Julia environment since it's much more space consuming. That's why I suggested a hybrid approach based on private_env package option. https://github.com/JuliaLang/Pkg.jl/issues/777#issuecomment-428058336 (added: read https://github.com/JuliaPy/Conda.jl/issues/123#issuecomment-423376252 first)

Furthermore, to obtain some degree of de-duplication so that it becomes reasonable to use separate conda environment for some Julia environments, we need https://github.com/JuliaLang/Pkg.jl/issues/777

[stevengj]

Using the Julia version will also lead to too many Conda installations; there is no strong reason not to share conda installations between Julia 1.0 and Julia 1.1, for example, or for that matter with Julia 0.7.

In the short run, I'm starting to feel like it will be better to just install conda in ~/.julia/conda (shared between all Julia versions and environments) and worry later about adding the option to let a Julia project/environment install its own conda virtualenv with a given set of packages. The latter seems like it depends on Pkg option support anyway.

[StefanKarpinski]

Yes, that seems pretty reasonable to me.