Get bash and git included in our Jupyter Lab installation

[agstephens]

How to test "bash" and "git" in Jupyter Lab installation

Ref: https://github.com/jupyterlab/jupyterlab-git

On a test server, install with:

yum remove -y git
yum -y install https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm
yum install -y git
conda install -c conda-forge jupyterlab jupyterlab-git

NOTE: We will need to above git version (from: packages.endpointdev.com).

We will need to add jupyterlab-git into this set of commands:

https://breezy.badc.rl.ac.uk/jasmin-notebooks/jasmin-notebooks/-/blob/master/images/notebook/Dockerfile#L38-46

Testing on a local installation

Then run, not as root, with:

jupyter lab # to run on localhost

#...or ...

jupyter lab --ip=192.168.50.81 # to run on a specific IP

Then, in the browser, you can select a terminal window and you can use the git drop-down and/or the git button on the left-hand panel.

[alaniwi]

Tested git integration locally. Used it to create this.

https://github.com/alaniwi/myproj/blob/cc73fbe673ccbde0f8737a3505c4862cdb570db0/mynb.ipynb

This includes a bash command example, but not a bash kernel.

[alaniwi]

Here is a notebook using a bash kernel

https://github.com/alaniwi/myproj/blob/4df180947e16548833b736b4400863128293f9c4/bash_notebook.ipynb

Added using:

pip install ipykernel
pip install bash_kernel
python -m bash_kernel.install

as shown at https://evodify.com/python-r-bash-jupyter-notebook/

This is all still just local testing so far - have not yet added to jasmin-notebooks repo. Will create a branch to try it.

[alaniwi]

(Note: ignoring bash kernel stuff - not what we want apparently, just need to ensure that a bash terminal option is available.)

contents of working example

A working local example, which gave git integration and bash terminal, was installed with the following packages.

git installation

This as root (command as shown above):

yum -y install https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm
yum install -y git

caused this RPM package to be installed:

git-2.36.0-1.ep7.x86_64

(plus a load of dep-installs, mostly perl)

conda installation

(in fact done under another username),

• This in a new environment (conda create -n myenv; conda activate myenv)

  conda install -c conda-forge jupyterlab jupyterlab-git

  caused these conda packages to be installed:

  jupyterlab=3.4.4=pyhd8ed1ab_0
  jupyterlab-git=0.37.1=pyhd8ed1ab_0
  jupyterlab_server=2.15.0=pyhd8ed1ab_0
  notebook=6.4.12

  (plus other dependencies, not including jupyterhub)

whereas current dockerfile has:

      jupyterhub=1.4.2
      jupyterlab=3.1.7
      notebook=6.4.3

If I also do conda install jupyterhub (although this wasn't needed for the working test) then this installs

jupyterhub=2.3.1=pyhd8ed1ab_0

solution to be tried

Given that nothing explicit was done to enable the bash terminal, it is probably just a case of the versions used. The fixed versions in the Dockerfile probably need updated.

Therefore trying with:

      jupyterhub=2.3.1
      jupyterlab=3.4.4
      jupyterlab-git=0.37.1
      notebook=6.4.12

[alaniwi]

How the local testing was done

(Includes some general instructions on using an ssh-enabled jaspy container, which I want to capture for use elsewhere, though should maybe be copied somewhere.)

Create a docker image which is jaspy + ssh

Here is the Dockerfile. The FROM line might need updating according to the tag applied to the Jaspy docker image.

FROM jaspy3.10-m3-4.9.2-r20220624
RUN yum install -y openssh-server xorg-x11-xauth
RUN groupadd sshgroup && useradd -ms /bin/bash -g sshgroup sshuser
RUN mkdir -p /home/sshuser/.ssh
COPY idkey.pub /home/sshuser/.ssh/authorized_keys
RUN chown sshuser:sshgroup /home/sshuser/.ssh/authorized_keys && chmod 600 /home/sshuser/.ssh/authorized_keys
RUN sed -i "s/^.*X11Forwarding.*$/X11Forwarding yes/" /etc/ssh/sshd_config
RUN sed -i "s/^.*X11UseLocalhost.*$/X11UseLocalhost no/" /etc/ssh/sshd_config 
RUN /usr/sbin/sshd-keygen /etc/ssh/ssh_host_rsa_key
EXPOSE 22
CMD ["/usr/sbin/sshd","-D"]

where idkey and idkey.pub exist in the directory with the Dockerfile after running: ssh-keygen -t rsa -f idkey

Create an image: docker build -t jaspy-ssh .

How to start container and connect to it

Start container with port forwarding of port 2022 on the physical host to port 22 on the container.

docker run --name jaspy-ssh-c -d -p 2022:22 jaspy-ssh

(using names jaspy-ssh for the image and jaspy-ssh-c for the container)

To connect as root (not using ssh):

docker exec -it jaspy-ssh-c /bin/bash

To ssh in as non-root:

ssh-keygen -R '[localhost]:2022'  # remove stale known-host entries
ssh -i idkey sshuser@localhost -p 2022

To activate jaspy (inside the container) - although not needed for this test because this test was using a different conda environment to host the notebook, writable by sshuser:

source /opt/conda/etc/profile.d/conda.sh
conda activate /opt/jaspy

How to do the notebook tests for this issue

• connect as root as shown above, and do the yum installation for git as shown above

• connect as the sshuser but with also a port forward on the ssh command in order to connect to the jupyterlab instance that we intend to run on port 8888 (this is the default jupyterlab port):

  ssh -AX -i idkey -p 2022 -L 8888:localhost:8888 sshuser@localhost

as the sshuser:

• source the conda.sh command as shown above, but instead of activating /opt/jaspy, create a new environment and activate it (see myenv above) - this gets stored under ~sshuser/.conda/env
• start the notebook using jupyter lab
• if a reminder of URL with token to connect to it is needed, type jupyter server list, should be of the form http://localhost:8888/lab?token=........

[alaniwi]

Working in pre-production on gitlab.

Tested both the github integration and the bash terminal.

Terminal is available either under File -> New -> Terminal, or from the launcher it is available under "other":