Building custom container for CUDA

[philschmid]

Hello,

Thank you for what you are building here! I sadly cannot use the existing containers, since we need to use the cuda ubuntu images. Therefore i was trying to create a customer container following the issue: https://stackoverflow.com/questions/68193812/micromamba-inside-docker-container and the documentation as well as looking into the code you provide. But i am still facing the micromamba not found issue.

Below is my dockerfile

FROM nvidia/cuda:11.7.0-devel-ubuntu22.04

WORKDIR /app

RUN apt-get update \
    && apt-get -y upgrade --only-upgrade systemd openssl cryptsetup \
    && apt-get install -y \
    bzip2 \
    curl \
    wget \
    git \
    git-lfs \
    tar \
    # audio
    libsndfile1-dev \
    ffmpeg \
    # install micromamba
    && wget -qO-  https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba \
    && touch /root/.bashrc \
    && ./bin/micromamba shell init -s bash -p /opt/conda  \
    && grep -v '[ -z "\$PS1" ] && return' /root/.bashrc  > /opt/conda/bashrc   # this line has been modified \
    && apt-get clean autoremove --yes \
    && rm -rf /var/lib/{apt,dpkg,cache,log}

RUN micromamba activate 

RUN micromamba install -y -n base -c conda-forge \
       pyopenssl=20.0.1 \
       python=${PYTHON_VERSION} \
       requests=2.25.1 && \
    micromamba clean --all --yes

ENTRYPOINT [ "python3" ]

[wholtz]

Hello @philschmid and thank you for your interest in micromamba.

Please see the documentation on activating a conda environment within a Dockerfile.

Another problem with your Dockerfile that you are going to run into is with your ENTRYPOINT .... Please see the documentation on activating a conda environment for an ENTRYPOINT.

That said, I just added two new base images to the micromamba image builds:

• nvidia/cuda:11.7.0-base-ubuntu22.04
• nvidia/cuda:11.7.0-base-ubuntu20.04

So the images derived from those might give you an easier path to the image you desire.

[philschmid]

Thank you for the pointers @wholtz. But i still see the /bin/sh: 1: micromamba: not found ->

FROM nvidia/cuda:11.7.0-devel-ubuntu22.04

WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
    && apt-get -y upgrade --only-upgrade systemd openssl cryptsetup \
    && apt-get install -y \
    bzip2 \
    curl \
    wget \
    git \
    git-lfs \
    tar \
    # audio
    libsndfile1-dev \
    ffmpeg \
    # install micromamba
    && wget -qO-  https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba \
    && touch /root/.bashrc \
    && ./bin/micromamba shell init -s bash -p /opt/conda  \
    && grep -v '[ -z "\$PS1" ] && return' /root/.bashrc  > /opt/conda/bashrc   # this line has been modified \
    && apt-get clean autoremove --yes \
    && rm -rf /var/lib/{apt,dpkg,cache,log}

ARG MAMBA_DOCKERFILE_ACTIVATE=1  
RUN micromamba install -y -n base -c conda-forge \
       pyopenssl=20.0.1 \
       python=${PYTHON_VERSION} \
       requests=2.25.1 && \
    micromamba clean --all --yes

full error

Step 9/11 : ARG MAMBA_DOCKERFILE_ACTIVATE=1
 ---> Running in 12201dd3b321
Removing intermediate container 12201dd3b321
 ---> 097f1bfc7edf
Step 10/11 : RUN micromamba install -y -n base -c conda-forge        pyopenssl=20.0.1        python=${PYTHON_VERSION}        requests=2.25.1 &&     micromamba clean --all --yes
 ---> Running in ba4ecbc0011f
/bin/sh: 1: micromamba: not found
The command '/bin/sh -c micromamba install -y -n base -c conda-forge        pyopenssl=20.0.1        python=${PYTHON_VERSION}        requests=2.25.1 &&     micromamba clean --all --yes' returned a non-zero code: 127

Its more like that the "bash" is not aware of the micromamba bin but it shouldn't it be added with the following to lines?

    && ./bin/micromamba shell init -s bash -p /opt/conda  \
    && grep -v '[ -z "\$PS1" ] && return' /root/.bashrc  > /opt/conda/bashrc   # this line has been modified \

[wholtz]

Sorry, I read your Dockerfile a bit too fast. You don't need:

ARG MAMBA_DOCKERFILE_ACTIVATE=1

but you do need to have micromamba in your $PATH.

However, I'm not sure why you are trying to re-invent the wheel here. I'd recommend doing this:

$ git clone https://github.com/mamba-org/micromamba-docker.git
#### GIT OUTPUT REMOVED ####
$ cd micromamba-docker/
$ docker build --build-arg=BASE_IMAGE=nvidia/cuda:11.7.0-base-ubuntu22.04 -t micromamba:cuda .
#### DOCKER BUILD OUTPUT REMOVED ####
$ echo 'FROM micromamba:cuda

RUN micromamba install -y -n base -c conda-forge \
       pyopenssl=20.0.1 \
       python=${PYTHON_VERSION} \
       requests=2.25.1 && \
    micromamba clean --all --yes' > Dockerfile.cuda
$ docker build -t my_cuda:01 -f Dockerfile.cuda .
#### DOCKER BUILD OUTPUT REMOVED ####
$ 

And be careful with that ${PYTHON_VERSION} in there, as I'm pretty sure it always is evaluating to an empty string, which defaults to the most recent version.

[philschmid]

I am not trying to reinvent the wheel, just to better understand how things are working. Also sadly cuda and cudnn version are changing quite frequently for DL Frameworks, e.g. Tensorflow currently supports cuda 11.2 and pytorch 11.6.

I solved it myself. Thats the current minimal dockerfile

FROM nvidia/cuda:11.7.0-devel-ubuntu22.04

WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
    && apt-get -y upgrade --only-upgrade systemd openssl cryptsetup \
    && apt-get install -y \
    bzip2 \
    curl \
    git \
    git-lfs \
    tar \
    # audio
    libsndfile1-dev \
    ffmpeg \
    && apt-get clean autoremove --yes \
    && rm -rf /var/lib/{apt,dpkg,cache,log}

# install micromamba
ENV MAMBA_ROOT_PREFIX=/opt/conda
ENV PATH=/opt/conda/bin:$PATH

RUN curl -L https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xj "bin/micromamba" \
    && touch /root/.bashrc \
    && ./bin/micromamba shell init -s bash -p /opt/conda  \
    && grep -v '[ -z "\$PS1" ] && return' /root/.bashrc  > /opt/conda/bashrc

RUN micromamba install -y -n base -c conda-forge \
       pyopenssl=20.0.1 \
       python=3.9.1 \
       requests=2.25.1 && \
    micromamba clean --all --yes