search

kleinee / jns

jupyter notebook and lab on Raspberry Pi
MIT License
235 stars 56 forks source link
bash julia jupyter python3 raspberry

readme

hardware os

python nodejs Julia R

jupyter piwheels

Jupyter Notebook & Lab Server on Raspberry Pi

Intro

Project Jupyter not only revolutionizes data-heavy research across domains - it also boosts personal productivity for problems on a much smaller scale. Due to openness it is an amazing platform for exploring concepts and learning new things.

I started setting up a Jupyter Notebook Server on a Raspberry Pi following this blog post by Arun Durvasula. Convinced of the potential of the platform I followed the development.

My personal exercise soon taught me a great deal about the underlying architcture. Given Jupyter's complexity, speed of growth and scale, it is remarkable that such a system runs fine on a Raspberry Pi.

This repository isn't really anything genuine: I owe big thanks to many contributors in the Jupyter, Raspberry Pi, Linux, Python, Julia and greater Open Source communities for providing all the beautiful building blocks - small and large.

What is new?

What do you need to follow along?

Installation

IMPORTANT NOTE on fresh installations

First boot with fresh SD card

sudo apt install -y git
git clone https://github.com/kleinee/jns
cd ~/jns/scripts

Technically you can now run sudo ./inst_jns.sh which is the installer script that combines the steps described below. If you follow along I assume that you run all scripts from inside the directory ~/jns/scripts.

You might not want all features on your system. Feel free to edit `inst_jns.sh' to suit your requirements.

Install required Raspbian packages with apt

sudo ./prep.sh

A couple of packages from the Raspbian repository are required during installation and later for a some Python packages to work properly. The script just fetches these packages and installs them.

#!/bin/bash
# script name:     prep.sh
# last modified:   2018/09/09
# sudo:            yes

script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

apt update && apt -y upgrade
apt -y install pandoc
apt -y install libxml2-dev libxslt-dev
apt -y install libblas-dev liblapack-dev
apt -y install libatlas-base-dev gfortran
apt -y install libtiff5-dev libjpeg62-turbo-dev
apt -y install zlib1g-dev libfreetype6-dev liblcms2-dev
apt -y install libwebp-dev tcl8.5-dev tk8.5-dev
apt -y install libharfbuzz-dev libfribidi-dev
apt -y install libhdf5-dev
apt -y install libnetcdf-dev
apt -y install python3-pip
apt -y install python3-venv
apt -y install libzmq3-dev
apt -y install sqlite3

Install required Python 3 packages with pip

./inst_stack.sh
#!/bin/bash
# script name:     inst_stack.sh
# last modified:   2018/01/14
# sudo: no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

if [ ! -d "$venv" ]; then
  python3 -m venv $env
fi

# activate virtual environment
source $env/bin/activate

pip3 install pip==9.0.0
pip3 install setuptools
pip3 install -U pip

cat requirements.txt | xargs -n 1 pip3 install

Configure Jupyter

./conf_jupyter.sh

With this script you generate a jupyter notebook configuration directory and in it a file called jupyter_notebook_config.py that holds the configuration settings for your notebook / lab server. You also create a folder notebooks in the home directory of user pi as the notebook_dir for your server. In the configuration file, you apply the following changes:

NOTE: This setup still uses password authentication. If you prefer token-based authentication, you have to change settings in the config file /home/pi/.jupyter/jupyter_notebook_config.py. Documentation of possible configuration settings can be found here.

After the basic configuration the script activates the bash kernel and activates extensions for Jupyter Notebook and JupyterLab. At the JupyterLab end this requires intstallation of node followed by installation of the underlying JS infrastructure which is a bit time-consuming but ultimately allows you to use ipywidgets, bqplot and potentially other extensions.

#!/bin/bash
# script name:     conf_jupyter.sh
# last modified:   2018/09/09
# sudo:            no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

# activate virtual environment
source $env/bin/activate

# generate config and create notebook directory
# if notebook directory exists, we keep it (-p)
# if configuration file exeists, we overwrite it (-y)

jupyter notebook -y --generate-config
cd $home
mkdir -p notebooks

target=~/.jupyter/jupyter_notebook_config.py

# set up dictionary of changes for jupyter_config.py
declare -A arr
app='c.NotebookApp'
arr+=(["$app.open_browser"]="$app.open_browser = False")
arr+=(["$app.ip"]="$app.ip ='*'")
arr+=(["$app.port"]="$app.port = 8888")
arr+=(["$app.enable_mathjax"]="$app.enable_mathjax = True")
arr+=(["$app.notebook_dir"]="$app.notebook_dir = '/home/pi/notebooks'")
arr+=(["$app.password"]="$app.password = 'sha1:5815fb7ca805:f09ed218dfcc908acb3e29c3b697079fea37486a'")

# apply changes to jupyter_notebook_config.py

for key in ${!arr[@]};do
    if grep -qF $key ${target}; then
        # key found -> replace line
        sed -i "/${key}/c ${arr[${key}]}" $target
    else
        # key not found -> append line
        echo "${arr[${key}]}" >> $target
    fi
done

# install bash kernel
python3 -m bash_kernel.install

# install extensions
jupyter serverextension enable --py jupyterlab
jupyter nbextension enable --py widgetsnbextension --sys-prefix
jupyter nbextension enable --py --sys-prefix bqplot

# activate clusters tab in notebook interface
/home/pi/.venv/jns/bin/ipcluster nbextension enable --user

# install nodejs and node version manager n
# if node is not yet installed
if which node > /dev/null
    then
        echo "node is installed, skipping..."
    else
        # install nodejs and node version manager n
        cd ~/jns
        # fix for issue #22
        # install nodejs and node version manager n
        # see: https://github.com/mklement0/n-install
        curl -L https://git.io/n-install | bash -s -- -y lts
fi

# install jupyter lab extensions
bash -i ./inst_lab_ext.sh

The script inst_lab_ext.sh - introduced by @Kevin--R to fix issue#23 has the following content:

#!/bin/bash
# script name:     inst_lab_ext.sh
# last modified:   2019/04/06
# sudo:            no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

. /home/pi/.bashrc
. $env/bin/activate
jupyter lab clean
jupyter labextension install @jupyter-widgets/jupyterlab-manager --no-build
jupyter labextension install bqplot --no-build
jupyter labextension install jupyterlab_bokeh --no-build
jupyter labextension install jupyter-leaflet --no-build
jupyter lab build

Start and access your server

Activate the virtual environment

Since you used a virtual environment to install Python modules, you need to activate this environment before you can start your server:

source /home/pi/.venv/jns/bin/activate

The prompt will change to indicate successfull activation preceding pi@hostname: with the envireonment name - in case pf this setup (jns). With hostname set to zerow it looks like this:

(jns) pi@zerow:~ $

Before you proceed

After installation completes, you will still need to activate the change made to ~\.bashrc when node was installed before doing anything that requires node.

You can be accomplish this by any of the following:

That's the reason for this warning during node installation:

  IMPORTANT: OPEN A NEW TERMINAL TAB/WINDOW or run `. /home/pi/.bashrc`
             before using n and Node.js.

You can see this by running the following commands after your installation completes:

pi@test-pi:~/jns $ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
pi@test-pi:~/jns $ . ~/.bashrc
pi@test-pi:~/jns $ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/home/pi/n/bin
pi@test-pi:~/jns $

If you look at your $PATH environment variable and see /home/pi/n/bin you are ready to use node.

Also note that if you uninstall node with n-uninstall /home/pi/n/bin will remain in your $PATH environment variable until you reboot or logout and log back in.

Start the server

To start your server just type jupyter notebook or jupyter lab

Access the server

To access your server form a webbrowser on a computer running on the same network as your Raspberry Pi, just open a browser and use the Pi's IP address / port 8888 as the url. 

xxx.xxx.xxx.xxx:8888

Change `xxx.xxx.xxx.xxx' to the IP address of the Raspberry Pi.

Login

During the configuration the default password for the server was set to jns. You can change this by typing:

(jns) pi@zerow:~ $ jupyter notebook password
Enter password:  ****
Verify password: ****

Install TeX (optional)

sudo ./inst_tex.sh
#!/bin/bash
# script name:     inst_tex.sh
# last modified:   2018/03/11
# sudo:            yes

script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

#------------------------------------------------------
apt install -y texlive-xetex
apt install -y latexmk
#------------------------------------------------------

Install Julia and the IJulia kernel (optional)

Alternative 1: Julia 1.1.0 (RECOMMENDED)

sudo ./inst_julia-1.1.0.sh

The Download Helper

As per comments in the script this is literally a 1:1 copy of code found on stack overflow - adjustments were only necessary to set the FILE_ID and the DESTINATION as required in the context of this repository. The helper is called by the installer script and is not meant to be exceuted manually.

#!/home/pi/.venv/jns/bin/python

#
# last modified 2019/05/26
#
# Python helper script to download Julia 1.1.0 binaries
# not meant to be executed manually
# https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url
#

FILE_ID = '1fj6pNAJgmUD7bsSXqh8ocC1wESx8jkRh'
DESTINATION = './julia-1.1.0-arm32bit.zip'

import requests

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)    

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

if __name__ == "__main__":
    file_id = FILE_ID
    destination = DESTINATION
    download_file_from_google_drive(file_id, destination)

The Installer

Note that the code assumes that julia is not present. 

#!/bin/bash
# script name:     inst_julia-1.1.0.sh
# last modified:   2019/05/26
# sudo:            yes

SCRIPT_NAME=$(basename -- "$0")
JNS_USER='pi'
HOME_DIR="/home/$JNS_USER"
ENV="$HOME_DIR/.venv/jns"

JULIA_HOME=$HOME_DIR/julia/

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$SCRIPT_NAME"
   exit 1
fi

#
# apt install dependencies
#
apt install -y build-essential 
apt install -y libatomic1
apt install -y gfortran 
apt install -y perl 
apt install -y wget
apt install -y m4
apt install -y cmake 
apt install -y pkg-config
apt install -y libopenblas-base libopenblas-dev
apt install -y libatlas3-base libatlas-base-dev
apt install -y liblapack-dev
apt install -y libmpfr-dev libgmp3-dev
apt install -y libgfortran3

#
# download and install julia based on architecture
#
su pi <<ONE
    cd $HOME_DIR
    . $ENV/bin/activate
    ./dnld_julia-1.1.0-arm32bit.py
    unzip ./julia-1.1.0-arm32bit.zip

    ARCHITECTURE=$(python -c 'import os; print(str(os.uname()[4]));')
    if (("$ARCHITECTURE" == "armv7l"))
    then
        mv ./julia1.1.0-arm32bit/rpi3/julia-1.1.0.zip $HOME_DIR
    else
        mv ./julia1.1.0-arm32bit/rpizero/julia-1.1.0.zip $HOME_DIR
    fi

    unzip julia-1.1.0.zip
    mv julia-1.1.0 julia
    rm -rf julia1.1.0-arm32bit
    rm ./julia-1.1.0-arm32bit.zip
    rm ./julia-1.1.0.zip
    rm -rf __MACOSX/
ONE

#
# add symbolic link for julia executable
#

ln -s $JULIA_HOME/bin/julia /usr/local/bin/julia

#
#  install IJulia kernel
#

su pi <<TWO
    julia -e 'using Pkg; Pkg.add("IJulia");'
    julia -e 'using IJulia;'
TWO

Alternative 2: Julia 0.6.0 provided with Raspbian Stretch (NOT RECOMMENDED)

This is NOT RECOMMENDED as Julia 0.6.0 is no longer maintained. I keep the installer here for reference only.

sudo ./inst_julia-0.6.0.sh
#!/bin/bash
# script name:     inst_julia-0.6.0.sh
# last modified:   2019/05/23
# sudo:            yes

env=/home/pi/.venv/jns
script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

env=/home/pi/.venv/jns

apt -y install julia

su pi <<EOF
source $env/bin/activate
julia -e 'Pkg.add("IJulia");'
julia -e 'using IJulia;'
EOF

Install R-3.6.0 and the IRkernel (optional)

Since the R binaries that come with Raspbian Stretch are quite dated, I decided to install R from source. Compilation takes a while. So be patient when running the script.

Note that this installer checks whether R is alredy present and if it is, skips compilition and just installs the IRkernel.

   sudo ./inst_R-3.6.0.sh
#!/bin/bash
# script name:     inst_R-3.6.0.sh
# last modified:   2019/05/19
# sudo:            yes

SCRIPT_NAME=$(basename -- "$0")
JNS_USER='pi'
HOME_DIR="/home/$JNS_USER"
ENV="$HOME_DIR/.venv/jns"

R_VERSION="R-3.6.0"
R_DOWNLOAD_URL="http://mirrors.psu.ac.th/pub/cran/src/base/R-3/$R_VERSION.tar.gz"
R_EXEC=$(which R)
R_HOME="$HOME_DIR/R"

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$SCRIPT_NAME"
   exit 1
fi

cd $HOME_DIR

#
#  apt install additional packages
#
apt install -y libreadline-dev
apt install -y libbz2-dev

#
#  download R source and compile
#  if R is not yet present  
#
su pi <<ONE
    if [ -z ${R_EXEC} ]; then
        if [-z ${R_HOME}]; then
            mkdir $R_HOME
        fi
        wget $R_DOWNLOAD_URL
        tar -xvf "$R_VERSION.tar.gz"
        rm "$R_VERSION.tar.gz"
        cd ./$R_VERSION 
        ./configure --with-x=no --disable-java --prefix=$R_HOME
        make && make install
        cd $HOME_DIR
        rm -rf $R_VERSION
    fi
ONE

#
#  create soft link in /usr/local/bin
#
ln -s $R_HOME/bin/R /usr/local/bin/R
ln -s $R_HOME/bin/Rscript /usr/local/bin/Rscript

su pi <<TWO
    . $ENV/bin/activate
    echo "install.packages('IRkernel', repos='http://cran.rstudio.com/')" | R --no-save
    echo "IRkernel::installspec()" | R --no-save
TWO

Install the SQLite kernel (optional)

./inst_sqlite.sh
#!/bin/bash
# script name:     inst_sqlite.sh
# last modified:   2018/09/09
# sudo:            no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

# activate virtual environment
source $env/bin/activate

# clone SQLite kernel repository
git clone https://github.com/brownan/sqlite3-kernel.git

# install kernel
cd sqlite3-kernel
python setup.py install
python -m sqlite3_kernel.install
cd ..
rm -rf sqlite3-kernel/

Install Python support for Raspberry Pi hardware (optional)

./inst_pi_hardware.sh

Setting up Python support for GPIO pins, the PICAMERA module and Sense HAT hardware in your virtual environment is almost as simple as you would commonly do without such environment.

#!/bin/bash
# script name:     inst_pi_hardware.sh
# last modified:   2018/10/04
# sudo: no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

# activate virtual environment
source $env/bin/activate

pip install RTIMULib
pip install sense-hat
pip install picamera
pip install gpiozero

## Install openCV (optional)

```bash
#!/bin/bash
# script name:     inst_opencv.sh
# last modified:   2018/11/27
# sudo:            yes

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

#------------------------------------------------------
apt install -y libjasper1 libjasper-dev
apt install -y libjpeg-dev libtiff5-dev libpng12-dev
apt install -y libilmbase12
apt install -y libopenexr22
apt install -y libgstreamer1.0-0
apt install -y libavcodec-extra57
apt install -y libavformat-dev
apt install -y libilmbase12
apt onstall -y libavcodec-dev
apt install -y libswscale-dev 
apt install -y libv4l-dev
apt install -y libgtk2.0-dev 
apt install -y libgtk-3-dev
apt install -y libxvidcore-dev 
apt install -y libx264-dev
#------------------------------------------------------

su - pi <<'EOF'
source /home/pi/.venv/jns/bin/activate
pip install opencv-python-headless
EOF

Start the server at boot with systemd (optional)

Credits for the following solution go to mt08xx:

To do this run:

sudo ./conf_service.sh

The file has the following content:

#!/bin/bash
# script name:     conf_service.sh
# last modified:   2018/09/09
# credits:         mt08xx
# sudo:            yes

script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

# create jupyter.sh in /home/pi and make it executable
cat << 'ONE' > /home/pi/jupyter_start.sh && chmod a+x /home/pi/jupyter_start.sh
#!/bin/bash
. /home/pi/.venv/jns/bin/activate
jupyter lab
#jupyter notebook
ONE

cat << 'TWO' | sudo tee /etc/systemd/system/jupyter.service
[Unit]
Description=Jupyter

[Service]
Type=simple
ExecStart=/home/pi/jupyter_start.sh
User=pi
Group=pi
WorkingDirectory=/home/pi/notebooks
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
TWO

# start jupyter
systemctl daemon-reload
systemctl start jupyter
systemctl enable jupyter
sudo systemctl stop jupyter

Put it all together

This script is just convenience - it executes the individual steps described above in the order necessary. Note that installation of additinal languages and their respective kernels as well as installtion of opnencv is deactivated by default as not all users may need this functionality. I recommend to run inst_jns.sh as is and install additional functionality using the individual scripts.

#!/bin/bash
# script name:     inst_jns.sh
# last modified:   2019/05/26
# sudo:            yes

script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

#-----------------------------------------------
# MANDATORY
#-----------------------------------------------

# make necessary preparations
./prep.sh

# install Python packages 
sudo -u pi ./inst_stack.sh

# configure server
sudo -u pi ./conf_jupyter.sh

#-----------------------------------------------
# OPTIONAL, RECOMMENDED
#-----------------------------------------------

# install TeX
./inst_tex.sh

# install support for Pi hardware
sudo -u pi ./inst_pi_hardware.sh

# set up service to start the server on boot
./conf_service.sh

#-----------------------------------------------
# OPTIONAL, DISABLED BY DEFAULT
#-----------------------------------------------

# install Julia 0.6.0 and the IJulia kernel NOT RECOMMENDED
# ./inst_julia-0.6.0.sh

# install Julia 1.1.0 and the IJulia kernel
# ./inst_julia-1.1.0.sh

# install R 3.6.0 and the IRkernel
# ./inst_R-3.6.0.sh

# install the SQLite3 kernel
# sudo -u pi ./inst_sqlite.sh

# install opencv
# ./inst_opencv.sh

Keep your installation up to date

Raspbian operating system

Python 3 packages