search

pangeo-data / jupyter-earth

Jupyter meets the Earth: combining research use cases in geosciences with technical developments within the Jupyter and Pangeo ecosystems.
https://jupytearth.org
Creative Commons Zero v1.0 Universal
28 stars 6 forks source link

Persistent Julia user state in the hub #118

Closed JordiBolibar closed 2 years ago

JordiBolibar commented 2 years ago

The way we are currently working with in Julia in the hub implies having to download and precompile all the packages each time we start a server. This is very time consuming, but it has other side effects such as having to configure paths each time. Indeed, one could add those in a script to be run at start time, but since we are developing a package for general usage we'd like to avoid relying on those sorts of tricks.

The main problem is that all the Julia packages and precompiled files are under /srv/julia. This folder is outside the user domain, so it is erased each time the server dies. Ideally, we would need those files to be inside /home/jovyan/.julia. Particularly, we would need persistence for the downloaded packages in /srv/julia/pkg/packages, and also the precompiled packages in /srv/julia/pkg/compiled. I'm not sure exactly which paths would be needed to be moved to each user, but probably everything inside /srv/julia/pkg.

Does this seem feasible from a technical point of view? This is how things are normally set up in local.

Thanks a lot in advance!

JordiBolibar commented 2 years ago

Today with @fperez and @facusapienza21 we investigated this issue and we came up with a solution. This is as simple as setting the JULIA_DEPOT_PATH environment variable to a local path in the user (e.g. home/jovyan/.julia). By adding the path (e.g. export JULIA_DEPOT_PATH="$HOME/.julia:) in the .bash_profile one can automatically set it up.

More details here: https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_DEPOT_PATH

consideRatio commented 2 years ago

@JordiBolibar ah excellent solution thanks for the follow up! I think for whoever does this, its good that the user is aware of the choice rather than its done by default as it may cause complexity if the Julia version is upgraded.

Hmmm. One may also consider installing ones own julia binary in a conda environment. Then one would be able to control the julia version - and I think any change to the version could force a re-compile of julia code. At the same time, having julia installed centrally and in a conda environment is probably a recipe for issues with PATH variables.

JordiBolibar commented 2 years ago

Yes, I think the current solution is quite satisfactory. Having Julia installed in a centralized manner ensures stability for all the users, but being able to store locally the packages and precompiled files is a must. So this fix provides this minimal part on the user side.

Let's see how it goes, and if we see that this needs some changes we can re-open this issue later on 😄