iot-salzburg / gpu-jupyter

GPU-Jupyter: Leverage the flexibility of Jupyterlab through the power of your NVIDIA GPU to run your code from Tensorflow and Pytorch in collaborative notebooks on the GPU.
Apache License 2.0
708 stars 235 forks source link

Suboptimal dockerfile structure in gpulibs #59

Closed nerok closed 3 years ago

nerok commented 3 years ago

I was looking into which parts of this image that mainly used space, as I felt the 17.9GB of uncompressed image (looked at cschranz/gpu-jupyter:v1.4_cuda-11.0_ubuntu-18.04). According to Docker desktop on my machine, these lines contributed around 6GB of space: https://github.com/iot-salzburg/gpu-jupyter/blob/d704653e3d8f816b583e1bb55fdcc2f2f1f9bdfb/src/Dockerfile.gpulibs#L11-L22

This differs a lot from how for instance the Jupyter Project uses Conda in their dockerfiles, such as in the base-notebook: https://github.com/jupyter/docker-stacks/blob/6d61708f1747d4fb15e3c0166805ebb5fba41ea1/base-notebook/Dockerfile#L134-L144

A better approach for the lines in gpulibs would be something like:

RUN conda install --quiet --yes \
    pyyaml mkl mkl-include setuptools cmake cffi typing && \
    pip install torch torchvision torchaudio torchviz && \
    conda clean --all -f -y && \
    fix-permissions $CONDA_DIR && \
    fix-permissions /home/$NB_USER

I tried using this diff on my machine and got a image size of 13.8GB (uncompressed) on the "v1.4_cuda-11.0_ubuntu-18.04" -branch

diff --git a/.build/Dockerfile b/.build/Dockerfile
index f20b066..596c051 100755
--- a/.build/Dockerfile
+++ b/.build/Dockerfile
@@ -307,15 +307,9 @@ RUN pip install --upgrade pip && \

 # Install PyTorch with dependencies
 RUN conda install --quiet --yes \
-    pyyaml mkl mkl-include setuptools cmake cffi typing
-
-# Check compatibility here:
-# https://pytorch.org/get-started/locally/
-# Installation via conda leads to errors installing cudatoolkit=10.1
-RUN pip install torch torchvision torchaudio torchviz
-
-# Clean installation
-RUN conda clean --all -f -y && \
+    pyyaml mkl mkl-include setuptools cmake cffi typing && \
+    pip install torch torchvision torchaudio torchviz && \
+    conda clean --all -f -y && \
     fix-permissions $CONDA_DIR && \
     fix-permissions /home/$NB_USER

Some similar fixes might be found for pip aswell.

If there is a reason why you have split this into multiple stages, there might be smart to add a comment about it.

mathematicalmichael commented 3 years ago

nice work! This seems a reasonable approach, feel free to open a PR. Doesn't look like it changes any functionality, just makes the image smaller, so I see nothing wrong with it.

nerok commented 3 years ago

Added a PR now, not sure who should be reviewing it.