Provide a tutorial for sysadmins for setting up shared packages

[arnavs]

Thanks @KristofferC for opening this. I could take a stab at this since we've been wrangling with it. And maybe if @jlperla thinks it's OK, we could link to the VSE Syzygy docker as an example.

[KristofferC]

That would be great!

[jlperla]

Definetely work taking the time to do a succinct write-up of the setup. But I am a little concerned about making links to the VSE syzygy setup... who knows how it might be merged or mutate with the other syzygy stuff. But quote whatever you want from it, and llink away to it on discourse, etc. where people don't expect permanence.

[arnavs]

No problem, we can take snippets or I can give a static fork. I just really want to make sure there’s code here for this.

[jlperla]

Try snippets. There might also be weird setup stuff for syzygy stuff that wouldn't apply to a normal jupyterhub installation. Neither of us have the docker chops to know, but I would worry that providing the whole thing might be more confusing that helpful.

[arnavs]

Something like this?

Creating Layered Environments

This is a brief tutorial for using Pkg3 in the context of a layered environment, or a situation where we provide a group of users with a common, immutable backend (say, a set of pre-installed packages), and allow them to make local changes as they see fit.

Install Julia

To do this programmatically, first make a directory where you want Julia to live (usually /opt/julia or /usr/bin/julia/, and then grab and unpack the Julia binary:

curl -s -L https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.0-linux-x86_64.tar.gz | \
     tar -C /opt/julia-1.0.0 -x -z --strip-components=1 -f -

This is happening as root in a Dockerfile.

Set up Shared User

First, we instantiate a shared user (here, jovyan), and give it a home directory. (The following are commands from a Dockerfile used to create this setup).

ENV NB_USER=jovyan
ENV HOME=/home/$NB_USER
USER $NB_USER

Instantiate that user's primary depot:

RUN mkdir -p $HOME/.julia/environments/v1.0 

(Note: A "depot" is something like a ~/.julia directory, or a directory with a packages/, registries/, environments/, compiled/, etc.)

Now, we want to install a set of packages. The cleanest way to do this is to grab TOML from online, and then use Pkg to instantiate.

RUN cd $HOME/.julia/environments/v1.0 \
# Grab the online TOML. 
&& wget -q https://raw.githubusercontent.com/QuantEcon/lecture-source-jl/master/notebooks/Manifest.toml -O Manifest.toml \
&& wget -q https://raw.githubusercontent.com/QuantEcon/lecture-source-jl/master/notebooks/Project.toml -O Project.toml

# Set up our environment. 
RUN julia -e "using Pkg; pkg\"build\"; pkg\"instantiate\"; pkg\"precompile\""

Clean Shared Environment

To avoid conflicts with resources in the individual user's home directory, it's important to scrub the shared backend of some objects:

• Registries: Registries aren't traversed in DEPOT_PATH order, but rather are merged and searched. There's also a known bug about multiple registries. So, we need to destroy the registry that came with our Julia install (rm -rf /opt/julia-1.0.0/local/share/julia/registries), and also the one that we cloned when installing our shared packages (rm -rf $HOME/.julia/registries).

• Environments: The only TOML should be living in the end user's directory, since we ideally treat the shared depot as simply a source of packages. To that end, we want to rm -rf $HOME/.Julia/environments. ⚠️ It's useful sometimes to "pre-seed" the end user with the Project and Manifest from the shared environment (so that their package names will automatically point to resources in the shared depot). If you want this, see the setup below:

  && rm -rf $HOME/.local \ 
  # Nuke the registry that came with Julia. 
  && rm -rf /opt/julia-1.0.0/local/share/julia/registries \ 
  # Nuke the registry that Jovyan uses. 
  && rm -rf $HOME/.julia/registries \ 
  && mkdir -p /home/jupyter/.julia/ \ 
  # Copy over the environments. 
  && cp -r $HOME/.julia/environments /home/jupyter/.julia/ \
  # Nuke the jovyan environment. 
  && rm -rf $HOME/.julia/environments 

⚠️ Make sure the end user has read/execute permissions on the shared environment; e.g.

RUN chmod -R go+rx /home/jovyan/.julia

Pre-Seeding

As above, if you want to initially seed the end user's TOML with the Project and Manifest from the shared depot (after all build steps), simply copy over the environments/ directory as above before destroying the shared environment.

Set the end user's DEPOT_PATH

Running something like the following

ENV JULIA_DEPOT_PATH="/home/jupyter/.julia:/home/jovyan/.julia:/opt/julia"

will ensure that the end user looks to the local directory for reading/writing/cloning new packages, but will search the shared environment before grabbing packages from the internet. The last entry /opt/julia is where we installed Julia itself.