Notes on setting The Littlest Jupyter Hub on AWS

[mslw]

For future reference, these are the notes from the TLJH on AWS setup I've done for a workshop with ~10 participants. AWS pricing is given following https://instances.vantage.sh (EU-central, as of today) and does not include storage. Follow up to #8

AWS setup

Follow the Installing on AWS guide on tljh's website

In point 6: t3a.small (price: $0.022/hr, $0.53/day) instance. In point 8: 32 GB of gp2

After setting up a password, establish an elastic IP address in AWS console, so that the same address will be assigned after stop / start.

Right before the workshop, the instance can be scaled from the AWS console. right click - instance settings - change instance type Possibly wanted (M - Most scenarios, ie. general purpose): m5a.2xlarge (8 vCPU, 32 GB RAM) $0.42/hr m5a.4xlarge (16 vCPU, 64 GB RAM) $0.83/hr

Jupyter Hub setup

Interface

Enable JupyterLab interface by default, following [[https://tljh.jupyter.org/en/latest/howto/env/notebook-interfaces.html][Change default UI for users]] from TLJH

sudo tljh-config set user_environment.default_app jupyterlab
sudo tljh-config reload hub

HTTPS

https://tljh.jupyter.org/en/latest/howto/admin/https.html

Software

sudo -E conda install -c conda-forge datalad
sudo -E conda install -c conda-forge pillow                 # for examples in module 1
sudo -E conda install -c conda-forge pandas seaborn pandoc  # for examples in module 4
sudo apt install tig tree
sudo apt install emacs

Other (possible, not done)

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html https://tljh.jupyter.org/en/latest/topic/idle-culler.html

[mslw]

For the workshop, I set the idle cull timeout to 90 minutes to avoid interruptions (and restored to 10 minutes when scaling back down to a smaller instance:

sudo tljh-config set services.cull.timeout 5400  # time in seconds
sudo tljh-config reload

Also, installed latex anyway, to be able to produce a pdf from pandoc. It's about 130 MB download and 640 MB of disk space used, so not exactly lightweight (therefore not putting it into the written examples). However, for some reason, I couldn't get the hub's html preview to display a local image (Markdown preview worked very well).

sudo apt install texlive texlive-plain-generic texlive-xetex

edit 2022-05-24 apparently also needs xelatex, added above

[mslw]

Here's an updated guide, after I had a little bit more fun with AWS / TLJH.

AWS setup: launch instance

The TLJH guide currently describes the old AWS interface. With the new one, things are even simpler.

0. In Services, find EC2 (stands for Elastic Compute Cloud), and click Launch Instances.
1. Name and tags: enter the name that will show in tour EC2 management console - something that will identify it for you, like "tljh-workshop".
2. Applications and OS Imahes: choose Ubuntu and pick Ubuntu Server 20.04 LTS. Note: Although the docs suggest 18.04, the hub builds and works correctly with Ubuntu 20.04 LTS and Ubuntu 22.04. It doesn't work with Debian (TLJH installer checks OS type at the beginning and straight out refuses if it's not Ubuntu).
3. Instance type: docs recommend t3.small, I went with t3a.medium. It can be easily upscaled later. See notes below.
4. Key pair (login): create new or use one that you created previously. This thing will be used for ssh access from your terminal (probably only needed if things go wrong, as you should be able to do everything via the Jupyter Hub browser interface).
5. Network settings: click edit and create security group allowing http (TCP port 80), https (TCP port 443) and ssh (TCP port 22), as per tljh docs point 10 (or use existing one, if you created such one previously)
6. Configure storage. 2GB for hub + storage per user times number of users. Depends how much stuff you want to install (that also requires storage) and have available. I went with a total of something around 30 GB gp2.
7. Advanced details. Expand, and in the very end paste the code provided in TLJH docs, replacing your username. The shebang is important, don't miss it!

   #!/bin/bash
   curl -L https://tljh.jupyter.org/bootstrap.py \
   | sudo python3 - \
   --admin <admin-user-name>

   Note. This bootstrap script is a convenience thing, so that the server is set up from the get-go and you'll work only through jupyter hub. If you don't paste this bit, you can SSH from your terminal and follow the steps described in "Installing on your own server" as if you worked on any other remote server.

8. Launch instance. Give it about 10 minutes to complete installation.

AWS: IP settings

We want our IP address to stay constant even if we stop our instance, and we want to assign a domain to our hub (TLJH's requirement for https access) so we need an "Elastic IP"

0. Go to Elastic IPs (also in EC2, under Network & Security heading in the left-hand-side panel in EC2 console)
1. Click the orange "Allocate Elastic IP addresses", pick the region that you want, keep "Amazon's pool of IPv4 addresses" selected, and click Allocate
2. Select the IP address that was allocated. Click "Actions" (next to the orange button), and select Associate Elastic IP address. Associate it with the instance that you've created.
3. Tell that address to your friendly admin and ask for it to be associated with an address within your institutional domain.

Jupyter Hub: post-launch config

This is a bit brief, but posted in hope that I annotate it a bit more later (or better - write a script that does that). Instructions include changing shell to zsh and doing some preconfiguration on the user side.

1. Set user_environment.default_app with tljh-config (see tljh docs)
2. Set https.enabled, https.letsencrypt (see tljh docs)
3. If you need to change password, when logged in, go to <your hub address>/hub/auth/change-password (undocumented feature)
4. apt update, apt upgrade, apt install zsh, sudo -E conda install git
5. git clone INM7's dotfiles and install them (or use your own)
6. Create a file .jupyter/jupyter_notebook_config.py and write:

   c.NotebookApp.terminado_settings = {"shell_command": ["/usr/bin/zsh"]}

   there. Re-launch your hub. This setting determines the shelled used by the Terminal in jupyterlab. Anything you set to SHELL environment variable won't matter, as it is checked on hub launch. This is the only way that worked for me (also: shells are considered to be chosen by the user, so makes sense to have this setting in user level). (tip from this thread)

7. To avoid having to git ignore ipynb checkpoints, create .gitignore_local and tell git to use it via git config --global core.excludesfile ~/.gitignore_global (tip from here)
8. Copy configuration files from your home to /etc/skel - contents of this directory will be placed in home directory for newly created users. You want: .dotfiles (if using INM7's collection), .gitconfig, .gitignore_global, and .jupyter/jupyter_notebook_config.py
9. Go to /etc/skel/ and symlink config files, if using INM7's dotfiles (if not, just place your own): ln -s .dotfiles/link/.zshrc .zshrc, and the same for .zlogin, .zprofile, and .zshenv
10. Apt / conda install whatever you need (see first 2 posts)
11. Only then add users. Maybe add a test one and see if all works.
12. Increase cull timeout with sudo tljh-config set services.cull.timeout <time in seconds> (you may wish to do this only before the workshop)

Notes on AWS instances

• The letter "a" in type name (eg. t3a.medium) means that they are running on AMD, and are slightly cheaper than their Intel counterparts (without the a).
• There are different instance types. We're interested in T3 and M5, both of which are multipurpose. T3 are burstable (meaning they accumulate credits when idling and use them when active, so don't offer full performance all the time, which suits us well). M5 are fixed-performance multipurpose (you get the full capacity all the time).
• TLJH recommends t3.small to start with. 2 vCPU and 2 GB RAM should be enough to install everything (though some conda installs may be pushing the 2GB RAM limit for some reason)
• I went with t3a.medium from the get-go. 2 vCPU and 4GB RAM are very comfortable for all set-up, and my intuition is that it should be fine for a few concurrent users if you want to leave the hub running between workshop days - price around $1 a day seems fine for that.
• For the workshop sessions, I upscaled to M5a.4xlarge (live sessions only, meaning downscaling back to t3a.medium overnight in a 2-day format). The "quadruple extra large" offers 16 vCPU and 64 GB RAM. Judging by looking at htop, this is serious overprovisioning even for ca 16 people, but at $1 an hour it seems a reasonable price for comfort.
• To scale (up or down), go to EC2 management console, select your instance, right click (or click on Actions), and stop it. Then again right-click (or Actions), instance settings, change instance type. Then bring it back up.
• Elastic IP incurs cost when not associated with a running instance, but it is marginal.
• After the workshop, you will wont to pull everything down, including storage (if left sitting, storage may generate a couple dollars per month).

[jsheunis]

More notes

User management

Depending on group size and logistics, one can create users beforehand via the admin control panel (e.g. using their email addresses, or the username part of the email), and let users create their own passwords once they navigate to the hub URL for the first time. Different authentication options are possible (e.g. admin can also authenticate users individually after they sign up).

Base setup for all users

TLJH creates the same base conda environment for all users. Admin can install standard packages for all users in that env using sudo -E followed by the installation commands (e.g. sudo -E conda install -c conda-forge datalad).

Shell config

The shell choice command (chsh) is a user-s[ecific setting and hard to enforce from admin-level. By the time a user accesses the shell in the hub it is already running with bash. chsh won't work other than for direct ssh connection. Setting the shell is therefore don at the admin level with .jupyter/jupyter_notebook_config.py in /etc/skel, of which the content is copied to each user on creation. After creating the .jupyter/jupyter_notebook_config.py file, the hub should be reset for it to take effect.

more...

[mslw]

As an alternative to inm7 dotfiles (git-cloneable from a public url) I was happy with pure and the following .zshrc:

zshrc content

``` # Set up the prompt fpath+=($HOME/.zsh/pure) autoload -Uz promptinit promptinit prompt pure setopt histignorealldups sharehistory # Use emacs keybindings even if our EDITOR is set to vi bindkey -e # Keep 1000 lines of history within the shell and save it to ~/.zsh_history: HISTSIZE=1000 SAVEHIST=1000 HISTFILE=~/.zsh_history # set LS_COLORS eval `dircolors -b` alias diff='diff --color=auto' alias grep='grep --color=auto' alias ls='ls --color=auto' # Use modern completion system autoload -Uz compinit compinit setopt extendedglob zstyle ':completion:*' completer _complete _ignored zstyle ':completion:*' matcher-list '' 'm:{[:lower:]}={[:upper:]}' 'm:{[:lower:][:upper:]}={[:upper:][:lower:]} r:|[._-]=* r:|=*' 'l:|=* r:|=*' # Show tab-completion menu, with colors zstyle ':completion:*' menu select zstyle ':completion:*:default' list-colors ${(s.:.)LS_COLORS} # Up arrow search autoload -U up-line-or-beginning-search autoload -U down-line-or-beginning-search zle -N up-line-or-beginning-search zle -N down-line-or-beginning-search bindkey "$key[Up]" up-line-or-beginning-search # Up bindkey "$key[Down]" down-line-or-beginning-search # Down # allow games (cowsay) path+=('/usr/games/') ```

I am using pure prompt on my own PC, and the above configuration is based on my own (partially autogenerated by zsh).

[jsheunis]

Extra notes for installing Singularity 3.10.4 (current latest) on the EC2 instance with Ubuntu 22.04 (also current latest available on EC2):

Ideal: install neurodebian package

The easiest would be to install the debian package singularity-container via neurodebian, but this seems to not be packaged yet for Ubuntu 22.04 (also check here if it is packaged for the architecture type of the instance). If it were packaged (or if you have an instance running an older version of Ubuntu), it would be straightforward to install with the instructions provided by neurodebian here: http://neuro.debian.net/install_pkg.html?p=singularity-container. You have to select the appropriate Ubuntu version, the download sever, and whether you want all packages or only those with guaranteed freedoms. E.g. for Ubuntu 20.04, installing all software from the server at Jülich:

wget -O- http://neuro.debian.net/lists/focal.de-fzj.full | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list
sudo apt-key adv --recv-keys --keyserver hkps://keyserver.ubuntu.com 0xA5D32F012649A5A9
sudo apt-get update
sudo -E apt-get install singularity-container

Another possible ideal that I saw too late: installing the debian package provided by Sylabs/singularity

See https://docs.sylabs.io/guides/3.10/admin-guide/installation.html#install-from-provided-rpm-deb-packages

Fallback: building from source

This is what I did.

First install dependencies:

# Ensure repositories are up-to-date
sudo apt-get update
# Install debian packages for dependencies
sudo apt-get install -y \
   build-essential \
   libseccomp-dev \
   libglib2.0-dev \
   pkg-config \
   squashfs-tools \
   cryptsetup \
   runc

Then install Go (the language in which Singularity is written). Download and extract an archive (I used version 1.19.3 and architecture amd64; you can see the available options here: https://go.dev/dl/), and add the location to the PATH variable:

export VERSION=1.19.3 OS=linux ARCH=amd64 && \  # Replace the values as needed
  wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \ # Downloads the required Go package
  sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \ # Extracts the archive
  rm go$VERSION.$OS-$ARCH.tar.gz    # Deletes the ``tar`` file

echo 'export PATH=/usr/local/go/bin:$PATH' >> ~/.zshrc && \
  source ~/.zshrc

Then download singularity from a release at https://github.com/sylabs/singularity/releases:

export VERSION=3.10.0 && # adjust this as necessary \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
    tar -xzf singularity-ce-${VERSION}.tar.gz && \
    cd singularity-ce-${VERSION}

and finally build it from source:

./mconfig && \
    make -C builddir && \
    sudo -E make -C builddir install

[mih]

I am closing this again. For a notes issue, the status "open" is not really needed from my POV.

[adswa]

Notes on setting up Singularity continued:

To use the github release, I did the following:

# download the deb file
wget https://github.com/sylabs/singularity/releases/download/v3.11.5/singularity-ce_3.11.5-focal_amd64.deb

install dependencies as specified on their website https://docs.sylabs.io/guides/3.10/admin-guide/installation.html#install-from-provided-rpm-deb-packages

sudo -E apt-get install -y \
   build-essential \
   libseccomp-dev \
   libglib2.0-dev \
   pkg-config \
   squashfs-tools \
   cryptsetup \
   runc

Install:

sudo -E apt install ./singularity-ce_3.11.5-focal_amd64.deb 

[adswa]

I previously had the issue that unprivileged users were not able to use datalad containers-run:

~/my-analysis master
❯ datalad containers-run -m "run classification analysis in python environment" \
  --container-name python-env \
  --input "input/sub-01/meg/sub-01_task-somato_meg.fif" \
  --output "figures/*" \
  "python3 code/mne_time_frequency_tutorial.py {inputs}"
[INFO   ] Making sure inputs are available (this may take some time) 
get(ok): input/sub-01/meg/sub-01_task-somato_meg.fif (file) [from s3-PUBLIC...]                                                                                
[INFO   ] == Command start (output follows) =====                                                                                                              
ERROR  : No setuid installation found, for unprivileged installation use: ./mconfig --without-suid
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /home/jupyter-tonka/my-analysis (dataset) [singularity exec .datalad/environments/p...]
action summary:
  get (notneeded: 2, ok: 1)
  run (error: 1)

I saw this with an installation from a 3.11.5 github release, and with a from-source build mirroring that of @jsheunis with Singularity version 3.10.0. I worked around this using the instruction in the error message, and rebuild from source using

./mconfig --without-suid && \
    make -C builddir && \
    sudo -E make -C builddir install

Other solutions I tried:

• I did not work to find the singularity config file, and just set the use_setuid option to "no".

[adswa]

I have spent a few hours exploring the possibilities of Singularity with a more recent Ubuntu (noble-nombat, 24.04). Sadly, I was not able to allow (non-privileged) users to run containers by any of the following means:

• a) build version 3.11.5 from source, using --without-suid
• b) Option a), plus setting "ALLOW SETUID" in /usr/local/singularity/singularity.conf to no
• c) Options a) and b), but with version 3.10
• d) deb file version 4.2.1.noble
• e) Option a) and b), but with version 4.2.0

While the issue with versions 3.x is that non-privileged users can't run containers, the problem with 4.x seems to also apply to admin users:

[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
INFO:    Mounting image with FUSE.
WARNING: squashfuse mount failed, falling back to extraction: failed to load image: invalid SIF magic
INFO:    Converting SIF file to temporary sandbox...
FATAL:   while preparing image: extraction failed: root filesystem extraction failed: extract command failed: ERROR  : Failed to create container process: Operation not permitted
: exit status 1
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /home/jupyter-adina/test (dataset) [singularity exec .datalad/environments/p...]

edit: I realized just after posing that the issue with 4.x likely is the container - all my demo containers are likely too old to be compatible with singularity 4.x, I believe they changed container formats.