FAST + Docker = <3 ?

[andreped]

Describe the bug I was trying to use FAST within a docker container, but it seems to fail due to some OpenCL issues, likely installation gone wrong or runPipeline requiring there to be a display to render the viewer.

To Reproduce Here is the extremely simple Dockerfile I made:

# create virtual ubuntu docker image
# FROM nvidia/opencl:latest
FROM ubuntu:22.04

# set language, format, and stuff
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# install pip and wget
RUN apt-get update -y && \
    apt-get install -y \
        python3-pip \
        wget

# Install OpenCL
RUN apt-get install intel-opencl-icd -y

# Install OpenGL
RUN apt-get install -y \
    libopengl0 \
    libgl1 \
    libusb-1.0-0-dev \
    libcairo2

# install python3 dependencies
RUN pip3 install --default-timeout=1000 pyfast==4.6.0

First create the Dockerfile above and build the docker image (from where the Dockerfile lies):

docker build -t h2gnet .

Then simply check if runPipeline can be called within the docker image by:

docker run -it --rm h2gnet:latest runPipeline --verbose

System:

• OS and version: Ubuntu 22.04 (but within Docker - tested with macOS 12 laptop)
• GPU and OpenCL vendor (NVIDIA, AMD, Intel): Intel

Expected behavior Should be able to use runPipeline on a headless machine, such as within docker. Even a machine without GPU (only Intel CPU for instance). Same for pyFAST which appear to have the same limitation.

Error log Produced from running the final command above:

INFO [140671336266496] Loaded configuration file: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin/fast_configuration.txt
INFO [140671336266496] Test data path: /root/FAST/data/
INFO [140671336266496] Kernel source path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//kernels/
INFO [140671336266496] Kernel binary path: /root/FAST//kernel_binaries/
INFO [140671336266496] Documentation path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//doc/
INFO [140671336266496] Pipeline path: /root/FAST//pipelines/
INFO [140671336266496] Qt plugins path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//plugins/
INFO [140671336266496] Library path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//lib/
INFO [140671336266496] Creating new QApp
WARNING [140671336266496] Unable to open X display. Disabling visualization.
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
INFO [140671336266496] QApp already exists..
ERROR [140671336266496] Terminated with unhandled exception: clGetPlatformIDs

[andreped]

Alternatively, you could try pyFAST by going into the built docker container:

docker run -it --rm h2gnet:latest

and then running:

python3 -c "import fast"

which in my case results in:

WARNING [139799543730176] Unable to open X display. Disabling visualization.
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/fast/__init__.py", line 22, in <module>
    fast.ImageFileImporter.create('') # Trigger splash, GL context initialization etc.
  File "/usr/local/lib/python3.10/dist-packages/fast/fast.py", line 13671, in create
    return _fast.ImageFileImporter_create(filename)
RuntimeError: clGetPlatformIDs

I then installed clinfo by:

apt-get install clinfo

from inside the docker container, and then run clinfo I got:

root@9d86e9f491eb:/# clinfo
Number of platforms                               0

If I then try to install:

apt install mesa-opencl-icd

I get that there is one platform. Trying pyFAST again it prompts:

WARNING [140231796592640] Unable to open X display. Disabling visualization.
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/fast/__init__.py", line 22, in <module>
    fast.ImageFileImporter.create('') # Trigger splash, GL context initialization etc.
  File "/usr/local/lib/python3.10/dist-packages/fast/fast.py", line 13671, in create
    return _fast.ImageFileImporter_create(filename)
RuntimeError: No valid OpenCL platforms!

[smistad]

Have you tried the Nvidia docker and installed Nvidia ooencl ICD?

[andreped]

Have you tried the Nvidia docker and installed Nvidia ooencl ICD?

Nvidia has a lot of different docker images. Do you mean this one nvidia/opencl. If so, yes, I have, same issue.

Regarding the nvidia-opencl-icd, I was unable to find it directly, but seems like I can install it if I choose a driver. I can give that a go.

Otherwise, there is the nvidia/opengl image, which is a lot newer than the OpenCL docker. I can try that next. But feels like I'm just doing all permutations now. If you were to set this up on a fresh ubuntu machine (without NVIDIA GPU) which docker image would you use and what would you install in addition (if necessary)?

[smistad]

https://linuxhandbook.com/setup-opencl-linux-docker/

[andreped]

Made a new Dockerfile according to the linuxhandbook you suggested above:

# build custom image from the ubuntu docker image
FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

# maintainer of docker file
# MAINTAINER André Pedersen <andre.pedersen@sintef.no>

# set language, format, and stuff
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# install dependencies and OpenCL
RUN apt-get update && apt-get -y upgrade \
  && apt-get install -y \
    apt-utils \
    unzip \
    tar \
    curl \
    xz-utils \
    ocl-icd-libopencl1 \
    opencl-headers \
    clinfo \
    python3-pip \
    ;
RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility

# Install OpenGL
RUN apt-get install -y \
    libopengl0 \
    libgl1 \
    libusb-1.0-0-dev \
    libcairo2

# install python3 dependencies
RUN pip3 install --default-timeout=1000 pyfast==4.6.0

Made minor adjustments to add OpenGL support and to install pyFAST. To reproduce do:

git clone https://github.com/andreped/H2G-Net.git
cd H2G-Net/src/docker/
docker build -t h2gnet .
docker run -it --rm h2gnet:latest runPipeline --verbose

Which results in the same issue as earlier (even on Ubuntu 18.04):

INFO [140383789644544] Loaded configuration file: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin/fast_configuration.txt
INFO [140383789644544] Test data path: /root/FAST/data/
INFO [140383789644544] Kernel source path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//kernels/
INFO [140383789644544] Kernel binary path: /root/FAST//kernel_binaries/
INFO [140383789644544] Documentation path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//doc/
INFO [140383789644544] Pipeline path: /root/FAST//pipelines/
INFO [140383789644544] Qt plugins path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//plugins/
INFO [140383789644544] Library path: /usr/local/lib/python3.8/dist-packages/fast/bin/../bin//..//lib/
INFO [140383789644544] Creating new QApp
WARNING [140383789644544] Unable to open X display. Disabling visualization.
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
INFO [140383789644544] QApp already exists..
ERROR [140383789644544] Terminated with unhandled exception: clGetPlatformIDs

Hence, there is still something that is not working. Perhaps the docker image is not able to access the resources from the local hardware. Not sure. Any ideas what I could try?

[andreped]

I manged to fix the issue by building the docker image from the NVIDIA CUDA image, where I set the CUDA version to match what I have where the docker image will be used.

In the end, this is the Dockerfile that worked with pyFAST:

# build custom docker image from the nvidia CUDA image
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04

# headless
ARG DEBIAN_FRONTEND=noninteractive

# set language, format, and stuff
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# install dependencies and OpenCL
RUN apt-get update && apt-get -y upgrade \
  && apt-get install -y \
    apt-utils \
    unzip \
    tar \
    curl \
    xz-utils \
    ocl-icd-libopencl1 \
    opencl-headers \
    clinfo \
    python3-pip \
    ;
RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility

# Install OpenGL
RUN apt-get install -y \
    libopengl0 \
    libgl1 \
    libusb-1.0-0-dev \
    libcairo2

# install python3 dependencies
RUN pip3 install --default-timeout=1000 pyfast==4.6.0

As this issue has been resolved, I'm closing this issue.

However, it could be a good idea to update the documentations to include how to use FAST with Docker, @smistad.