pip install packages were lost when loading image

[huangnengCSU]

Version of Singularity What version of Singularity are you using? Run:

$ singularity --version

singularity-3.7.3

Describe the bug A clear and concise description of what the bug is.

When I build a new image, I install a python packages like this:

%post
git clone https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
cd Ranger-Deep-Learning-Optimizer && pip install -e .

Then I enter the image and check the pip environment, I see:

ranger                 0.1.dev0  /tools/Ranger-Deep-Learning-Optimizer

Then I push the image to library and load the image from a new server. When I check the pip list, I found the installed ranger package is missing. When I build an image with docker and push it to dockerhub. Then I pull the image with docker and ranger package can be found in the image. Then I pull the image from dockerhub with singularity, the ranger package can not be found.

To Reproduce Steps to reproduce the behavior:

Bootstrap: library
From: ubuntu:16.04
Stage: build

%post
    mkdir -p /tools/
    apt-get update
    apt-get install -y wget git vim make gcc g++ parallel libncurses5-dev zlib1g-dev libbz2-dev liblzma-dev tabix tar bzip2
    cd /tools/
    wget https://github.com/samtools/samtools/releases/download/1.15.1/samtools-1.15.1.tar.bz2
    tar -jxvf samtools-1.15.1.tar.bz2 && cd samtools-1.15.1 && ./configure && make && make install
    cd /tools/
    wget https://github.com/samtools/bcftools/releases/download/1.15.1/bcftools-1.15.1.tar.bz2
    tar -jxvf bcftools-1.15.1.tar.bz2 && cd bcftools-1.15.1 && ./configure && make && make install
    cd /tools/
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    bash Miniconda3-latest-Linux-x86_64.sh -b -p /usr/local/miniconda3/
    export PATH=/usr/local/miniconda3/bin:$PATH
    conda config --add channels bioconda
    conda config --add channels conda-forge
    conda install pytorch-cpu
    pip install torchnet torchmetrics pyyaml pandas tqdm tensorboardx matplotlib tables pysam
    python -m pip install git+https://github.com/lessw2020/Ranger21.git
    cd /tools/
    git clone https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
    cd Ranger-Deep-Learning-Optimizer && pip install -e .
%environment
    export PATH=/usr/local/miniconda3/bin:$PATH
    export PATH=/usr/local/bin:$PATH

Expected behavior A clear and concise description of what you expected to happen. I hope the ranger package and ranger21 package can be found when I load the image from a new server.

OS / Linux Distribution Which Linux distribution are you using?

$ cat /etc/os-release

NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

Installation Method Write here how you installed SingularityCE. Eg. RPM, source.

source

Additional context Anything else which might be relevant. E.g. if the bug only occurs on a specific filesystem, or kernel version etc.

[dtrudg]

This is unlikely to be a bug, but is due to a difference in default behavior between Singularity and Docker.

By default, Singularity passes host environment variables, the user's home directory, etc. into the container for convenience. However, this commonly causes issues with containers that have Python tools, on hosts where a user has setup python environments outside of the container. Python looks in user home directories for packages, and can have the package path influenced from the environment. Try running the container with --containall or --compat to see if you are then able to see the package. See also this section of the user guide:

https://sylabs.io/guides/3.9/user-guide/singularity_and_docker.html#differences-and-limitations-vs-docker

Also, when you build containers using conda for python, and install pip packages into it, you should generally create and activate a conda environment fully, instead of just setting PATH. This can help avoid issues as it better isolates the installation and is recommended in the conda documentation:

Issues may arise when using pip and conda together. When combining conda and pip, it is best to use an isolated conda environment.

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#pip-in-env https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#installing-non-conda-packages

[huangnengCSU]

To dtrudg: Thanks for your response. I followed your solution by running the container with --containall, then the python packages are all found. Best Neng