search

ReproNim / neurodocker

Generate custom Docker and Singularity images, and minimize existing containers
https://www.repronim.org/neurodocker/
Apache License 2.0
326 stars 97 forks source link

How to activate conda environment that was created using a .yml file? #506

Open JohannesWiesner opened 1 year ago

JohannesWiesner commented 1 year ago

We would like to create a new conda environment using a .yml file. With our current script we end up with a container, where neither the base nor our custom "yml"-environment seem to be activated. E.g.:

(csp_wiesner_johannes) johannes.wiesner@pf41jell:~$ docker run -t -i --rm -p 8888:8888 new_neurodocker:test
csp@a00e2ac887bf:/$ conda env list
# conda environments:
#
base                     /opt/miniconda-latest
csp                      /opt/miniconda-latest/envs/csp

In old neurodocker versions, there was the activate=True option. I always thought this option did exactly that (activating a custom environment). On top of that, I could come up with a script (very close to an old version of generate_docker.sh from the nipype_tutorial) that uses 1.) activate=True and 2.) --env LD_LIBRARY_PATH="/opt/miniconda-latest/envs/csp:$LD_LIBRARY_PATH" and 3.) --run-bash "source activate csp". Here the activation seem to have worked (I am just not sure which of the three options / or their combination is responsible for that):

(csp_wiesner_johannes) johannes.wiesner@pf41jell:~$ docker run -t -i --rm -p 8888:8888 old_neurodocker:test
csp@397f016452b3:~$ conda env list
# conda environments:
#
base                     /opt/miniconda-latest
csp                   *  /opt/miniconda-latest/envs/csp

csp@397f016452b3:~$

Pasting both the old/new bash-scripts/Dockerfiles here:

Latest script:

Note that a .yml file has to be present in the directory that the script is in. You can execute this script by running bash generate_dockerfile.sh test_env.yml. For example use test_env.yml as provided in our repo

if [ -n "$1" ]; then
    conda_yml_file=$1
    echo "Using the provided .yml file"
else
    python ./tcy/tcy.py csp docker linux --ignore_yml_name --no_pip_requirements_file --tsv_path ./tcy/packages.tsv --yml_dir .
    conda_yml_file=environment.yml
    echo "Using the .yml file as generated with the tcy submodule"
fi

echo "Adding yaml-file ${conda_yml_file} to be installed in conda. Make sure that the .yml file does not contain a name nor a prefix"

# function to create a dockerfile
generate_docker() {
    docker run -i --rm repronim/neurodocker:0.9.4 generate docker \
        --base-image neurodebian:stretch-non-free \
        --arg DEBIAN_FRONTEND='noninteractive' \
        --pkg-manager apt \
        --install opts="--quiet" \
            gcc \
            g++ \
            octave \
        --spm12 \
            version=r7771 \
        --freesurfer \
            version=7.1.1 \
        --copy $conda_yml_file /tmp/ \
        --miniconda \
            version=latest \
            yaml_file=/tmp/$conda_yml_file \
            env_name=csp \
        --user csp \
        --run 'mkdir /home/csp/data && chmod 777 /home/csp/data && chmod a+s /home/csp/data' \
        --run 'mkdir /home/csp/output && chmod 777 /home/csp/output && chmod a+s /home/csp/output' \
        --run 'mkdir /home/csp/code && chmod 777 /home/csp/code && chmod a+s /home/csp/code' \
        --run 'mkdir /home/csp/.jupyter && echo c.NotebookApp.ip = \"0.0.0.0\" > home/csp/.jupyter/jupyter_notebook_config.py'
}

# generate Dockerfile
generate_docker > Dockerfile

Latest Dockerfile:

# Generated by Neurodocker and Reproenv.

FROM neurodebian:stretch-non-free
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update -qq \
           && apt-get install -y --quiet \
                  g++ \
                  gcc \
                  octave \
           && rm -rf /var/lib/apt/lists/*
ENV FORCE_SPMMCR="1" \
    SPM_HTML_BROWSER="0" \
    SPMMCRCMD="/opt/spm12-r7771/run_spm12.sh /opt/matlab-compiler-runtime-2010a/v713 script" \
    LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/opt/matlab-compiler-runtime-2010a/v713/runtime/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/bin/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/sys/os/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/extern/bin/glnxa64" \
    MATLABCMD="/opt/matlab-compiler-runtime-2010a/v713/toolbox/matlab"
RUN export TMPDIR="$(mktemp -d)" \
    && apt-get update -qq \
    && apt-get install -y -q --no-install-recommends \
           bc \
           ca-certificates \
           curl \
           libncurses5 \
           libxext6 \
           libxmu6 \
           libxpm-dev \
           libxt6 \
           multiarch-support \
           unzip \
    && rm -rf /var/lib/apt/lists/* \
    && _reproenv_tmppath="$(mktemp -t tmp.XXXXXXXXXX.deb)" \
    && curl -fsSL --retry 5 -o "${_reproenv_tmppath}" http://mirrors.kernel.org/debian/pool/main/libx/libxp/libxp6_1.0.2-2_amd64.deb \
    && apt-get install --yes -q "${_reproenv_tmppath}" \
    && rm "${_reproenv_tmppath}" \
    && apt-get update -qq \
    && apt-get install --yes --quiet --fix-missing \
    && rm -rf /var/lib/apt/lists/* \
    && echo "Downloading MATLAB Compiler Runtime ..." \
    && curl -fL -o "$TMPDIR/MCRInstaller.bin" https://dl.dropbox.com/s/zz6me0c3v4yq5fd/MCR_R2010a_glnxa64_installer.bin \
    && chmod +x "$TMPDIR/MCRInstaller.bin" \
    && "$TMPDIR/MCRInstaller.bin" -silent -P installLocation="/opt/matlab-compiler-runtime-2010a" \
    && rm -rf "$TMPDIR" \
    && unset TMPDIR \
    # Install spm12
    && echo "Downloading standalone SPM12 ..." \
    && curl -fL -o /tmp/spm12.zip https://www.fil.ion.ucl.ac.uk/spm/download/restricted/utopia/previous/spm12_r7771_R2010a.zip \
    && unzip -q /tmp/spm12.zip -d /tmp \
    && mkdir -p /opt/spm12-r7771 \
    && mv /tmp/spm12/* /opt/spm12-r7771/ \
    && chmod -R 777 /opt/spm12-r7771 \
    && rm -rf /tmp/spm* \
    # Test
    && /opt/spm12-r7771/run_spm12.sh /opt/matlab-compiler-runtime-2010a/v713 quit
ENV OS="Linux" \
    PATH="/opt/freesurfer-7.1.1/bin:/opt/freesurfer-7.1.1/fsfast/bin:/opt/freesurfer-7.1.1/tktools:/opt/freesurfer-7.1.1/mni/bin:$PATH" \
    FREESURFER_HOME="/opt/freesurfer-7.1.1" \
    FREESURFER="/opt/freesurfer-7.1.1" \
    SUBJECTS_DIR="/opt/freesurfer-7.1.1/subjects" \
    LOCAL_DIR="/opt/freesurfer-7.1.1/local" \
    FSFAST_HOME="/opt/freesurfer-7.1.1/fsfast" \
    FMRI_ANALYSIS_DIR="/opt/freesurfer-7.1.1/fsfast" \
    FUNCTIONALS_DIR="/opt/freesurfer-7.1.1/sessions" \
    FS_OVERRIDE="0" \
    FIX_VERTEX_AREA="" \
    FSF_OUTPUT_FORMAT="nii.gz# mni env requirements" \
    MINC_BIN_DIR="/opt/freesurfer-7.1.1/mni/bin" \
    MINC_LIB_DIR="/opt/freesurfer-7.1.1/mni/lib" \
    MNI_DIR="/opt/freesurfer-7.1.1/mni" \
    MNI_DATAPATH="/opt/freesurfer-7.1.1/mni/data" \
    MNI_PERL5LIB="/opt/freesurfer-7.1.1/mni/share/perl5" \
    PERL5LIB="/opt/freesurfer-7.1.1/mni/share/perl5"
RUN apt-get update -qq \
    && apt-get install -y -q --no-install-recommends \
           bc \
           ca-certificates \
           curl \
           libgomp1 \
           libxmu6 \
           libxt6 \
           perl \
           tcsh \
    && rm -rf /var/lib/apt/lists/* \
    && echo "Downloading FreeSurfer ..." \
    && mkdir -p /opt/freesurfer-7.1.1 \
    && curl -fL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/7.1.1/freesurfer-linux-centos6_x86_64-7.1.1.tar.gz \
    | tar -xz -C /opt/freesurfer-7.1.1 --owner root --group root --no-same-owner --strip-components 1 \
         --exclude='average/mult-comp-cor' \
         --exclude='lib/cuda' \
         --exclude='lib/qt' \
         --exclude='subjects/V1_average' \
         --exclude='subjects/bert' \
         --exclude='subjects/cvs_avg35' \
         --exclude='subjects/cvs_avg35_inMNI152' \
         --exclude='subjects/fsaverage3' \
         --exclude='subjects/fsaverage4' \
         --exclude='subjects/fsaverage5' \
         --exclude='subjects/fsaverage6' \
         --exclude='subjects/fsaverage_sym' \
         --exclude='trctrain'
COPY ["test_env.yml", \
      "/tmp/"]
ENV CONDA_DIR="/opt/miniconda-latest" \
    PATH="/opt/miniconda-latest/bin:$PATH"
RUN apt-get update -qq \
    && apt-get install -y -q --no-install-recommends \
           bzip2 \
           ca-certificates \
           curl \
    && rm -rf /var/lib/apt/lists/* \
    # Install dependencies.
    && export PATH="/opt/miniconda-latest/bin:$PATH" \
    && echo "Downloading Miniconda installer ..." \
    && conda_installer="/tmp/miniconda.sh" \
    && curl -fsSL -o "$conda_installer" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && bash "$conda_installer" -b -p /opt/miniconda-latest \
    && rm -f "$conda_installer" \
    && conda update -yq -nbase conda \
    # Prefer packages in conda-forge
    && conda config --system --prepend channels conda-forge \
    # Packages in lower-priority channels not considered if a package with the same
    # name exists in a higher priority channel. Can dramatically speed up installations.
    # Conda recommends this as a default
    # https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html
    && conda config --set channel_priority strict \
    && conda config --system --set auto_update_conda false \
    && conda config --system --set show_channel_urls true \
    # Enable `conda activate`
    && conda init bash \
    && conda env create  --name csp --file /tmp/test_env.yml \
    # Clean up
    && sync && conda clean --all --yes && sync \
    && rm -rf ~/.cache/pip/*
RUN test "$(getent passwd csp)" \
    || useradd --no-user-group --create-home --shell /bin/bash csp
USER csp
RUN mkdir /home/csp/data && chmod 777 /home/csp/data && chmod a+s /home/csp/data
RUN mkdir /home/csp/output && chmod 777 /home/csp/output && chmod a+s /home/csp/output
RUN mkdir /home/csp/code && chmod 777 /home/csp/code && chmod a+s /home/csp/code
RUN mkdir /home/csp/.jupyter && echo c.NotebookApp.ip = \"0.0.0.0\" > home/csp/.jupyter/jupyter_notebook_config.py

# Save specification to JSON.
USER root
RUN printf '{ \
  "pkg_manager": "apt", \
  "existing_users": [ \
    "root" \
  ], \
  "instructions": [ \
    { \
      "name": "from_", \
      "kwds": { \
        "base_image": "neurodebian:stretch-non-free" \
      } \
    }, \
    { \
      "name": "arg", \
      "kwds": { \
        "key": "DEBIAN_FRONTEND", \
        "value": "noninteractive" \
      } \
    }, \
    { \
      "name": "install", \
      "kwds": { \
        "pkgs": [ \
          "gcc", \
          "g++", \
          "octave" \
        ], \
        "opts": "--quiet" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "apt-get update -qq \\\\\\n    && apt-get install -y --quiet \\\\\\n           g++ \\\\\\n           gcc \\\\\\n           octave \\\\\\n    && rm -rf /var/lib/apt/lists/*" \
      } \
    }, \
    { \
      "name": "env", \
      "kwds": { \
        "FORCE_SPMMCR": "1", \
        "SPM_HTML_BROWSER": "0", \
        "SPMMCRCMD": "/opt/spm12-r7771/run_spm12.sh /opt/matlab-compiler-runtime-2010a/v713 script", \
        "LD_LIBRARY_PATH": "$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/opt/matlab-compiler-runtime-2010a/v713/runtime/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/bin/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/sys/os/glnxa64:/opt/matlab-compiler-runtime-2010a/v713/extern/bin/glnxa64", \
        "MATLABCMD": "/opt/matlab-compiler-runtime-2010a/v713/toolbox/matlab" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "export TMPDIR=\\"$\(mktemp -d\)\\"\\napt-get update -qq\\napt-get install -y -q --no-install-recommends \\\\\\n    bc \\\\\\n    ca-certificates \\\\\\n    curl \\\\\\n    libncurses5 \\\\\\n    libxext6 \\\\\\n    libxmu6 \\\\\\n    libxpm-dev \\\\\\n    libxt6 \\\\\\n    multiarch-support \\\\\\n    unzip\\nrm -rf /var/lib/apt/lists/*\\n_reproenv_tmppath=\\"$\(mktemp -t tmp.XXXXXXXXXX.deb\)\\"\\ncurl -fsSL --retry 5 -o \\"${_reproenv_tmppath}\\" http://mirrors.kernel.org/debian/pool/main/libx/libxp/libxp6_1.0.2-2_amd64.deb\\napt-get install --yes -q \\"${_reproenv_tmppath}\\"\\nrm \\"${_reproenv_tmppath}\\"\\napt-get update -qq\\napt-get install --yes --quiet --fix-missing\\nrm -rf /var/lib/apt/lists/*\\necho \\"Downloading MATLAB Compiler Runtime ...\\"\\ncurl -fL -o \\"$TMPDIR/MCRInstaller.bin\\" https://dl.dropbox.com/s/zz6me0c3v4yq5fd/MCR_R2010a_glnxa64_installer.bin\\nchmod +x \\"$TMPDIR/MCRInstaller.bin\\"\\n\\"$TMPDIR/MCRInstaller.bin\\" -silent -P installLocation=\\"/opt/matlab-compiler-runtime-2010a\\"\\nrm -rf \\"$TMPDIR\\"\\nunset TMPDIR\\n# Install spm12\\necho \\"Downloading standalone SPM12 ...\\"\\ncurl -fL -o /tmp/spm12.zip https://www.fil.ion.ucl.ac.uk/spm/download/restricted/utopia/previous/spm12_r7771_R2010a.zip\\nunzip -q /tmp/spm12.zip -d /tmp\\nmkdir -p /opt/spm12-r7771\\nmv /tmp/spm12/* /opt/spm12-r7771/\\nchmod -R 777 /opt/spm12-r7771\\nrm -rf /tmp/spm*\\n# Test\\n/opt/spm12-r7771/run_spm12.sh /opt/matlab-compiler-runtime-2010a/v713 quit" \
      } \
    }, \
    { \
      "name": "env", \
      "kwds": { \
        "OS": "Linux", \
        "PATH": "/opt/freesurfer-7.1.1/bin:/opt/freesurfer-7.1.1/fsfast/bin:/opt/freesurfer-7.1.1/tktools:/opt/freesurfer-7.1.1/mni/bin:$PATH", \
        "FREESURFER_HOME": "/opt/freesurfer-7.1.1", \
        "FREESURFER": "/opt/freesurfer-7.1.1", \
        "SUBJECTS_DIR": "/opt/freesurfer-7.1.1/subjects", \
        "LOCAL_DIR": "/opt/freesurfer-7.1.1/local", \
        "FSFAST_HOME": "/opt/freesurfer-7.1.1/fsfast", \
        "FMRI_ANALYSIS_DIR": "/opt/freesurfer-7.1.1/fsfast", \
        "FUNCTIONALS_DIR": "/opt/freesurfer-7.1.1/sessions", \
        "FS_OVERRIDE": "0", \
        "FIX_VERTEX_AREA": "", \
        "FSF_OUTPUT_FORMAT": "nii.gz# mni env requirements", \
        "MINC_BIN_DIR": "/opt/freesurfer-7.1.1/mni/bin", \
        "MINC_LIB_DIR": "/opt/freesurfer-7.1.1/mni/lib", \
        "MNI_DIR": "/opt/freesurfer-7.1.1/mni", \
        "MNI_DATAPATH": "/opt/freesurfer-7.1.1/mni/data", \
        "MNI_PERL5LIB": "/opt/freesurfer-7.1.1/mni/share/perl5", \
        "PERL5LIB": "/opt/freesurfer-7.1.1/mni/share/perl5" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "apt-get update -qq\\napt-get install -y -q --no-install-recommends \\\\\\n    bc \\\\\\n    ca-certificates \\\\\\n    curl \\\\\\n    libgomp1 \\\\\\n    libxmu6 \\\\\\n    libxt6 \\\\\\n    perl \\\\\\n    tcsh\\nrm -rf /var/lib/apt/lists/*\\necho \\"Downloading FreeSurfer ...\\"\\nmkdir -p /opt/freesurfer-7.1.1\\ncurl -fL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/7.1.1/freesurfer-linux-centos6_x86_64-7.1.1.tar.gz \\\\\\n| tar -xz -C /opt/freesurfer-7.1.1 --owner root --group root --no-same-owner --strip-components 1 \\\\\\n  --exclude='"'"'average/mult-comp-cor'"'"' \\\\\\n  --exclude='"'"'lib/cuda'"'"' \\\\\\n  --exclude='"'"'lib/qt'"'"' \\\\\\n  --exclude='"'"'subjects/V1_average'"'"' \\\\\\n  --exclude='"'"'subjects/bert'"'"' \\\\\\n  --exclude='"'"'subjects/cvs_avg35'"'"' \\\\\\n  --exclude='"'"'subjects/cvs_avg35_inMNI152'"'"' \\\\\\n  --exclude='"'"'subjects/fsaverage3'"'"' \\\\\\n  --exclude='"'"'subjects/fsaverage4'"'"' \\\\\\n  --exclude='"'"'subjects/fsaverage5'"'"' \\\\\\n  --exclude='"'"'subjects/fsaverage6'"'"' \\\\\\n  --exclude='"'"'subjects/fsaverage_sym'"'"' \\\\\\n  --exclude='"'"'trctrain'"'"'" \
      } \
    }, \
    { \
      "name": "copy", \
      "kwds": { \
        "source": [ \
          "test_env.yml", \
          "/tmp/" \
        ], \
        "destination": "/tmp/" \
      } \
    }, \
    { \
      "name": "env", \
      "kwds": { \
        "CONDA_DIR": "/opt/miniconda-latest", \
        "PATH": "/opt/miniconda-latest/bin:$PATH" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "apt-get update -qq\\napt-get install -y -q --no-install-recommends \\\\\\n    bzip2 \\\\\\n    ca-certificates \\\\\\n    curl\\nrm -rf /var/lib/apt/lists/*\\n# Install dependencies.\\nexport PATH=\\"/opt/miniconda-latest/bin:$PATH\\"\\necho \\"Downloading Miniconda installer ...\\"\\nconda_installer=\\"/tmp/miniconda.sh\\"\\ncurl -fsSL -o \\"$conda_installer\\" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh\\nbash \\"$conda_installer\\" -b -p /opt/miniconda-latest\\nrm -f \\"$conda_installer\\"\\nconda update -yq -nbase conda\\n# Prefer packages in conda-forge\\nconda config --system --prepend channels conda-forge\\n# Packages in lower-priority channels not considered if a package with the same\\n# name exists in a higher priority channel. Can dramatically speed up installations.\\n# Conda recommends this as a default\\n# https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html\\nconda config --set channel_priority strict\\nconda config --system --set auto_update_conda false\\nconda config --system --set show_channel_urls true\\n# Enable `conda activate`\\nconda init bash\\nconda env create  --name csp --file /tmp/test_env.yml\\n# Clean up\\nsync && conda clean --all --yes && sync\\nrm -rf ~/.cache/pip/*" \
      } \
    }, \
    { \
      "name": "user", \
      "kwds": { \
        "user": "csp" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "mkdir /home/csp/data && chmod 777 /home/csp/data && chmod a+s /home/csp/data" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "mkdir /home/csp/output && chmod 777 /home/csp/output && chmod a+s /home/csp/output" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "mkdir /home/csp/code && chmod 777 /home/csp/code && chmod a+s /home/csp/code" \
      } \
    }, \
    { \
      "name": "run", \
      "kwds": { \
        "command": "mkdir /home/csp/.jupyter && echo c.NotebookApp.ip = \\\\\\"0.0.0.0\\\\\\" > home/csp/.jupyter/jupyter_notebook_config.py" \
      } \
    } \
  ] \
}' > /.reproenv.json
USER csp
# End saving to specification to JSON.

Old script uses kaczmarj/neurodocker:0.7.0:

generate_docker() {
  docker run --rm kaczmarj/neurodocker:0.7.0 generate docker \
           --base neurodebian:stretch-non-free \
           --pkg-manager apt \
           --install apt_opts='--quiet' octave \
           --user=csp \
           --workdir '/home/csp' \
           --miniconda \
             conda_install="python=3.7 pandas jupyter jupyterlab" \
             pip_install="nilearn" \
             create_env="csp" \
             activate=True \
           --env LD_LIBRARY_PATH="/opt/miniconda-latest/envs/csp:$LD_LIBRARY_PATH" \
           --run-bash "source activate csp"
}

generate_docker > Dockerfile

Old Dockerfile:

# Generated by: Neurodocker version 0.7.0+0.gdc97516.dirty
# Latest release: Neurodocker version 0.9.4
# Timestamp: 2023/01/19 16:42:46 UTC
# 
# Thank you for using Neurodocker. If you discover any issues
# or ways to improve this software, please submit an issue or
# pull request on our GitHub repository:
# 
#     https://github.com/ReproNim/neurodocker

FROM neurodebian:stretch-non-free

USER root

ARG DEBIAN_FRONTEND="noninteractive"

ENV LANG="en_US.UTF-8" \
    LC_ALL="en_US.UTF-8" \
    ND_ENTRYPOINT="/neurodocker/startup.sh"
RUN export ND_ENTRYPOINT="/neurodocker/startup.sh" \
    && apt-get update -qq \
    && apt-get install -y -q --no-install-recommends \
           apt-utils \
           bzip2 \
           ca-certificates \
           curl \
           locales \
           unzip \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
    && dpkg-reconfigure --frontend=noninteractive locales \
    && update-locale LANG="en_US.UTF-8" \
    && chmod 777 /opt && chmod a+s /opt \
    && mkdir -p /neurodocker \
    && if [ ! -f "$ND_ENTRYPOINT" ]; then \
         echo '#!/usr/bin/env bash' >> "$ND_ENTRYPOINT" \
    &&   echo 'set -e' >> "$ND_ENTRYPOINT" \
    &&   echo 'export USER="${USER:=`whoami`}"' >> "$ND_ENTRYPOINT" \
    &&   echo 'if [ -n "$1" ]; then "$@"; else /usr/bin/env bash; fi' >> "$ND_ENTRYPOINT"; \
    fi \
    && chmod -R 777 /neurodocker && chmod a+s /neurodocker

ENTRYPOINT ["/neurodocker/startup.sh"]

RUN apt-get update -qq \
    && apt-get install -y --quiet \
           octave \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN test "$(getent passwd csp)" || useradd --no-user-group --create-home --shell /bin/bash csp
USER csp

WORKDIR /home/csp

ENV CONDA_DIR="/opt/miniconda-latest" \
    PATH="/opt/miniconda-latest/bin:$PATH"
RUN export PATH="/opt/miniconda-latest/bin:$PATH" \
    && echo "Downloading Miniconda installer ..." \
    && conda_installer="/tmp/miniconda.sh" \
    && curl -fsSL --retry 5 -o "$conda_installer" https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && bash "$conda_installer" -b -p /opt/miniconda-latest \
    && rm -f "$conda_installer" \
    && conda update -yq -nbase conda \
    && conda config --system --prepend channels conda-forge \
    && conda config --system --set auto_update_conda false \
    && conda config --system --set show_channel_urls true \
    && sync && conda clean -y --all && sync \
    && conda create -y -q --name csp \
    && conda install -y -q --name csp \
           "python=3.7" \
           "pandas" \
           "jupyter" \
           "jupyterlab" \
    && sync && conda clean -y --all && sync \
    && bash -c "source activate csp \
    &&   pip install --no-cache-dir  \
             "nilearn"" \
    && rm -rf ~/.cache/pip/* \
    && sync \
    && sed -i '$isource activate csp' $ND_ENTRYPOINT

ENV LD_LIBRARY_PATH="/opt/miniconda-latest/envs/csp:"

RUN bash -c 'source activate csp'

RUN echo '{ \
    \n  "pkg_manager": "apt", \
    \n  "instructions": [ \
    \n    [ \
    \n      "base", \
    \n      "neurodebian:stretch-non-free" \
    \n    ], \
    \n    [ \
    \n      "install", \
    \n      [ \
    \n        "apt_opts=--quiet", \
    \n        "octave" \
    \n      ] \
    \n    ], \
    \n    [ \
    \n      "user", \
    \n      "csp" \
    \n    ], \
    \n    [ \
    \n      "workdir", \
    \n      "/home/csp" \
    \n    ], \
    \n    [ \
    \n      "miniconda", \
    \n      { \
    \n        "conda_install": [ \
    \n          "python=3.7", \
    \n          "pandas", \
    \n          "jupyter", \
    \n          "jupyterlab" \
    \n        ], \
    \n        "pip_install": [ \
    \n          "nilearn" \
    \n        ], \
    \n        "create_env": "csp", \
    \n        "activate": true \
    \n      } \
    \n    ], \
    \n    [ \
    \n      "env", \
    \n      { \
    \n        "LD_LIBRARY_PATH": "/opt/miniconda-latest/envs/csp:" \
    \n      } \
    \n    ], \
    \n    [ \
    \n      "run_bash", \
    \n      "source activate csp" \
    \n    ] \
    \n  ] \
    \n}' > /neurodocker/neurodocker_specs.json
JohannesWiesner commented 1 year ago

Note: It is currently possible to post-hoc activate the custom environment by executing source activate csp once the container is running. It is not possible to activate the environment in the running container using conda activate csp. This will produce:

csp@5b5b1344459e:/$ conda activate csp

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.
JohannesWiesner commented 1 year ago

2nd note: Even after adding 

       --env LD_LIBRARY_PATH="/opt/miniconda-latest/envs/csp:$LD_LIBRARY_PATH" \
       --run-bash "source activate csp"

to the latest script it still does not work. Which makes me think that it must have something to do with the different neurodocker versions (kaczmarj/neurodocker:0.7.0 vs. repronim/neurodocker:0.9.4) or with the fact that the older version still had the activate=True option which somehow did the job. So the following lines will also not activate any env:

generate_docker() {
    docker run -i --rm repronim/neurodocker:0.9.4 generate docker \
        --base-image neurodebian:stretch-non-free \
        --arg DEBIAN_FRONTEND='noninteractive' \
        --pkg-manager apt \
        --install opts="--quiet" \
            gcc \
            g++ \
            octave \
        --spm12 \
            version=r7771 \
        --freesurfer \
            version=7.1.1 \
        --copy $conda_yml_file /tmp/ \
        --miniconda \
            version=latest \
            yaml_file=/tmp/$conda_yml_file \
            env_name="csp" \
        --env LD_LIBRARY_PATH="/opt/miniconda-latest/envs/csp:$LD_LIBRARY_PATH" \
        --run-bash "source activate csp" \
        --user csp \
        --run 'mkdir /home/csp/data && chmod 777 /home/csp/data && chmod a+s /home/csp/data' \
        --run 'mkdir /home/csp/output && chmod 777 /home/csp/output && chmod a+s /home/csp/output' \
        --run 'mkdir /home/csp/code && chmod 777 /home/csp/code && chmod a+s /home/csp/code' \
        --run 'mkdir /home/csp/.jupyter && echo c.NotebookApp.ip = \"0.0.0.0\" > home/csp/.jupyter/jupyter_notebook_config.py'
}
JohannesWiesner commented 1 year ago

3rd note: After checking the differences in Dockerfile when executing the old script with activate=True and without it, I found out that activate=True is responsible for adding these two lines of code to the Dockerfile which apparently do the magic:

-    && sync \
-    && sed -i '$isource activate csp' $ND_ENTRYPOINT

Why is activate=True not available anymore in newer neurodocker versions? Could we re-introduce it?

JohannesWiesner commented 1 year ago

@kaczmarj : I found a workaround for now by using --run 'echo source activate name_of_the_env >> /home/use_name/.bashrc'.

Example (make sure that conda_yml_file is a path to a .yml file of your choice):

generate_docker() {
    docker run -i --rm repronim/neurodocker:0.9.4 generate docker \
        --base-image neurodebian:stretch-non-free \
        --yes \
        --pkg-manager apt \
        --install opts="--quiet" \
            gcc \
            g++ \
            octave \
        --spm12 \
            version=r7771 \
        --freesurfer \
            version=7.1.1 \
        --copy $conda_yml_file /tmp/ \
        --miniconda \
            version=latest \
            yaml_file=/tmp/$conda_yml_file \
            env_name=csp \
        --user csp \
        --run 'mkdir /home/csp/data && chmod 777 /home/csp/data && chmod a+s /home/csp/data' \
        --run 'mkdir /home/csp/output && chmod 777 /home/csp/output && chmod a+s /home/csp/output' \
        --run 'mkdir /home/csp/code && chmod 777 /home/csp/code && chmod a+s /home/csp/code' \
        --run 'mkdir /home/csp/.jupyter && echo c.NotebookApp.ip = \"0.0.0.0\" > home/csp/.jupyter/jupyter_notebook_config.py' \
        --workdir /home/csp/code \
        --run 'echo source activate csp >> /home/csp/.bashrc'
}

I will close #16 in our repository for now (though you might find useful links there). I currently just follow this stackoverflow post by user merv who in general seems to know a lot of stuff about conda/Docker (helped me already a couple of times in the past on SO).

General takeaways:

  1. conda activate (although generally recommended over source activate) doesn't really do what we want.
  2. A good approach might be to edit the ./bashrc or .profile files

A possible solution for this issue:

Reintroduce --activate=true + introduce a way for users to specify which environment they would like to activate for which users (could be achieved by providing a tuple? Something like --activate "csp" "john.doe jane.doe"?)

djarecka commented 1 year ago

@kaczmarj @satra - looks like activate=True was removed in #378 . Do you see right now any reason to not bring it back?

kaczmarj commented 1 year ago

we could bring it back but there are some sharp edges. we would have to test that we are activating the environment correctly. the conda command will not be available in a /bin/sh shell in the docker image during build, so conda activate ENV would not work on its own. also the shell is started fresh in every RUN instruction, so activating the environment would not persist to the next RUN layer.

JohannesWiesner commented 1 year ago

I guess the only way to achieve this to either hardcode it to some configuration file like .bashrc or /etc/profile.d/activate_conda.sh (which would be more elegant because this ensures that the environment would be activated for all users) or to use an ENTRYPOINT so the first thing that happens, when the container is executed, is that the conda environment is activated. This is tricky though because one has to ensure that the container keeps running, see: https://stackoverflow.com/questions/41741895/docker-unable-to-start-an-interactive-shell-if-the-image-has-an-entry-script

I already tried the first approach which I find more elegant but somehow this doesn't work:

    --run 'touch /etc/profile.d/conda.sh' \
    --run "echo '#!/bin/bash' >> /etc/profile.d/conda.sh" \
    --run 'echo source activate csp >> /etc/profile.d/conda.sh' \

Using an ENTRYPOINT would also mean that you cannot use it for something else then. Afaik there can only be one ENTRYPOINT for each Dockerfile.

JohannesWiesner commented 1 year ago

Note: One advantage of using an entrypoint would be that the conda environment could then also be used when you use docker run -u $(id -u):$(id -g). Passing the user flag is necessary when you don't have root privileges on your host system. In this case you would want to use the -u flag to make sure that the host user can manipulate / delete files that were created by a container process. When you use this flag, you don't have home directory inside the container, so there's also no .bashrc file (or any of the other options) that you could write conda activate env_name into. However, so far I coulnd't create a bash script that can activate a conda environment. See https://github.com/conda/conda/issues/7980#issuecomment-1523630369

JohannesWiesner commented 1 year ago

This is tricky though because one has to ensure that the container keeps running, see: https://stackoverflow.com/questions/41741895/docker-unable-to-start-an-interactive-shell-if-the-image-has-an-entry-script

This can be solved by adding /usr/bin/env bash at the end of the bash script (I searched through an old version of neurodocker and found this solution). So the structure of the script should be:

#!/bin/bash

#. . . Do whatever you want here . . .

# we want the container to keep running after running the code above so we run this
/usr/bin/env bash
kimsin98 commented 1 year ago

Note: It is currently possible to post-hoc activate the custom environment by executing source activate csp once the container is running. It is not possible to activate the environment in the running container using conda activate csp.

This is happening because conda init was ran as root, not as csp, during docker build. Running --miniconda recipe as non-root user used to be possible, but not after the refactor.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 1 year ago

This issue was closed because it has been inactive for 14 days since being marked as stale.

github-actions[bot] commented 10 months ago

This issue is stale because it has been open for 30 days with no activity.