bootstraping is failing on 2.3

[DeepHorizons]

Version of Singularity:

Working: 2.2.1 Broken: 2.3 Gets farther: development branch

Expected behavior

Installing packages succeeds.

Actual behavior

Installing some packages fails. apt has several complaints,

• W: No sandbox user '_apt' on the system, can not drop privileges
• chown: invalid user: 'root:root'
• chown: invalid user: 'root:staff'

Installing sudo fails on 2.3, and emacsen-common fails on the development branch.

Running the development branch, bootstrapping one file seems to work, then bootstrapping a second file fails.

Steps to reproduce behavior

singularity create -s 4096 image.img
singularity bootstrap image.img base-notebook.def
singularity bootstrap image.img minimal-notebook.def

base-notebook.def (Will fail on 2.3, succeed on development)

BootStrap: docker
From: debian:stretch

%setup
    # commands to be executed on host outside container during bootstrap

    # There are some files here, but they don't conflict with anything

%post
    # commands to be executed inside container during bootstrap

    # add universe repo to apt sources and install some packages
    export DEBIAN_FRONTEND=noninteractive
    #sed -i '/deb/ s/$/ universe/' /etc/apt/sources.list
    sed -i "s/http:\/\/us.archive.ubuntu.com/http:\/\/mirrors.rit.edu/g" /etc/apt/sources.list
    # Delete any old cache
    apt-get clean
    rm -rf /var/lib/apt/lists/*
    apt-get update && apt-get install -yq --no-install-recommends \
        wget \
        bzip2 \
        ca-certificates \
        sudo \
        locales \
        fonts-liberation

    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
        locale-gen

    # Install Tini

    # Configure environment
    echo '\nexport CONDA_DIR=/opt/conda' >> /environment
    export CONDA_DIR=/opt/conda
    echo 'export PATH=$CONDA_DIR/bin:$PATH' >> /environment
    export PATH=$CONDA_DIR/bin:$PATH
    echo 'export SHELL=/bin/bash' >> /environment
    export SHELL=/bin/bash
    echo 'export NB_USER=jovyan' >> /environment
    export NB_USER=jovyan
    echo 'export NB_UID=1000' >> /environment
    export NB_UID=1000
    echo 'export HOME=/home/$NB_USER' >> /environment
    export HOME=/home/$NB_USER
    echo 'export LC_ALL=en_US.UTF-8' >> /environment
    export LC_ALL=en_US.UTF-8
    echo 'export LANG=en_US.UTF-8' >> /environment
    export LANG=en_US.UTF-8
    echo 'export LANGUAGE=en_US.UTF-8' >> /environment
    export LANGUAGE=en_US.UTF-8

    # Create jovyan user with UID=1000 and in the 'users' group

    # Setup jovyan home directory

    # Install conda as jovyan
    cd /tmp
    mkdir -p $CONDA_DIR
    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh
    echo "c59b3dd3cad550ac7596e0d599b91e75d88826db132e4146030ef471bb434e9a *Miniconda3-4.2.12-Linux-x86_64.sh" | sha256sum -c -
    /bin/bash Miniconda3-4.2.12-Linux-x86_64.sh -f -b -p $CONDA_DIR
    rm Miniconda3-4.2.12-Linux-x86_64.sh
    $CONDA_DIR/bin/conda config --system --add channels conda-forge
    $CONDA_DIR/bin/conda config --system --set auto_update_conda false
    conda clean -tipsy

    # Install Jupyter Notebook and Hub
    conda install --quiet --yes \
        'notebook=5.0.*' \
        'jupyterhub=0.7.*' \
        'jupyterlab=0.18.*' \
        && conda clean -tipsy

%runscript
    # commands to be executed when the container runs

    start-notebook.sh

%test
    # commands to be executed within container at close of bootstrap process

minimal-notebook.def (Fails on development branch)

%post
    # commands to be executed inside container during bootstrap
    . /environment

    addgroup staff

    # Install all OS dependencies for fully functional notebook server
    apt-get update && apt-get install -yq --no-install-recommends \
    git \
    vim \
    jed \
    emacs \
    build-essential \
    python-dev \
    unzip \
    libsm6 \
    pandoc \
    texlive-latex-base \
    texlive-latex-extra \
    texlive-fonts-extra \
    texlive-fonts-recommended \
    texlive-generic-recommended \
    texlive-xetex \
    lmodern \
    libxrender1 \
    inkscape

    # Install ipywidgets
    conda install --quiet --yes \
        'ipywidgets=5.1*' && \
        conda clean -tipsy

%runscript
    # commands to be executed when the container runs

    # This starts a bash shell if no arguments were passed, otherwise runs the command
    if [ -z "$1" ]
    then
        exec bash
    else
        exec "$@"
    fi

Here is a sample log from 2.3 from the base-notebook.def

INFO:root:Setting up sudo (1.8.16-0ubuntu1.4) ...
INFO:root:chown: invalid user: 'root:root'
INFO:root:dpkg: error processing package sudo (--configure):
INFO:root: subprocess installed post-installation script returned error exit status 1
INFO:root:Setting up fonts-liberation (1.07.4-1) ...
INFO:root:Errors were encountered while processing:
INFO:root: sudo
INFO:root:W: No sandbox user '_apt' on the system, can not drop privileges
INFO:root:E: Sub-process /usr/bin/dpkg returned an error code (1)
INFO:root:ABORT: Aborting with RETVAL=255

[gmkurtzer]

At the moment, the 2.3 release and the development branch are almost identical. I don't understand why one would work and not the other. Can you confirm that you installed both/either into a clean environment?

If it is still an issue, can you make a minimal bootstrap definition file, as simple as possible that reproduces this, and I (and maybe others) can try to run it.

Thanks!

[vsoch]

hey @DeepHorizons - it looks like you are just mirroring this container, almost to a T:

https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile

Why not just bootstrap that Docker image?

[vsoch]

I noticed in your examples you were trying to run files that exist in the original repo, but I'm bnot sure how they would be present in the image unless you downloaded them. So I adjusted the Singularity file a bit, and this worked perfect for me:

BootStrap: docker
From: debian:stretch

%labels
AUTHOR https://github.com/jupyter/docker-stacks/tree/master/base-notebook

%environment    
export CONDA_DIR=/opt/conda
export PATH=$CONDA_DIR/bin:$PATH
export SHELL=/bin/bash
export NB_USER=jovyan
export NB_UID=1000
export HOME=/home/$NB_USER
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

%post

    # add universe repo to apt sources and install some packages
    DEBIAN_FRONTEND=noninteractive
    export DEBIAN_FRONTEND
    #sed -i '/deb/ s/$/ universe/' /etc/apt/sources.list
    sed -i "s/http:\/\/us.archive.ubuntu.com/http:\/\/mirrors.rit.edu/g" /etc/apt/sources.list
    # Delete any old cache
    apt-get clean
    rm -rf /var/lib/apt/lists/*
    apt-get update && apt-get install -yq --no-install-recommends \
        wget \
        bzip2 \
        ca-certificates \
        locales \
        fonts-liberation

    echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
        locale-gen

    # Install Tini
    export CONDA_DIR=/opt/conda
    export PATH=$CONDA_DIR/bin:$PATH
    export SHELL=/bin/bash
    export NB_USER=jovyan
    export NB_UID=1000
    export HOME=/home/$NB_USER
    export LC_ALL=en_US.UTF-8
    export LANG=en_US.UTF-8
    export LANGUAGE=en_US.UTF-8

    # Install conda as jovyan
    cd /tmp
    mkdir -p $CONDA_DIR
    wget --quiet https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh
    echo "c59b3dd3cad550ac7596e0d599b91e75d88826db132e4146030ef471bb434e9a *Miniconda3-4.2.12-Linux-x86_64.sh" | sha256sum -c -
    /bin/bash Miniconda3-4.2.12-Linux-x86_64.sh -f -b -p $CONDA_DIR
    rm Miniconda3-4.2.12-Linux-x86_64.sh
    $CONDA_DIR/bin/conda config --system --add channels conda-forge
    $CONDA_DIR/bin/conda config --system --set auto_update_conda false
    conda clean -tipsy

    # Install Jupyter Notebook and Hub
    conda install --quiet --yes \
        'notebook=5.0.*' \
        'jupyterhub=0.7.*' \
        'jupyterlab=0.18.*' \
        && conda clean -tipsy

    wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/start.sh
    mv start.sh /usr/local/bin
    chmod u+x /usr/local/bin/start.sh

%runscript    
    exec /bin/bash /usr/local/bin/start.sh jupyter notebook "$@"

and then to generate:

singularity create --size 6000 conda.img
sudo singularity bootstrap conda.img Singularity

and to run you have to bind to the run directory (to get write as a user)

singularity run -B /tmp:/run/user conda.img

whala, notebook! And opening up showed all the crap in my /tmp directory, lol. Again, you could probably just have bootstrapped that docker image that had already done this.

[DeepHorizons]

@vsoch I am mirroring the container, but I'm not basing it off of debian. I've built a custom base image that we use on our cluster here at RIT that enables GPU support among other things. I then overlay the notebook on that. The debian bootstrap was just for example. As for the files, I downloaded the files and I add them in in the %setup step. I omitted them for brevity. Thanks for the example, nice to see %labels and %environment working!

I did more experimentation and was able to reproduce the bug consistently. It seems to be a problem with overlaying.

Steps to reproduce

1. singularity create -s 1024 image.img
2. singularity bootstrap image.img test1.def
3. singularity bootstrap image.img test2.def

test1.def

BootStrap: docker
From: ubuntu:16.04

test2.def

%post
    apt-get update && apt-get install -yq --no-install-recommends \
        sudo \
        emacsen-common \
        fontconfig-config \
        tex-common

All those installs fail with some sort of invalid user error. If any of those packages are moved into the first definition file, they install just fine.

[gmkurtzer]

I've reproduced this and know why it is happening. Fixing, standby...

[gmkurtzer]

Please see the fix in the development branch, and let me know if that helps!

Thanks!

[DeepHorizons]

That fixed it, thanks @gmkurtzer!