Docker file won´t built a working image

[monocilindro]

I cannot build the docker image using the docker file located in the root folder. It seems that the last 2 lines, besides uncommenting, need changing "conda run" for "RUN conda". But after it and when trying to build the docker image from get docker file, I am getting this error

executor failed running [conda run -n kluster_test /bin/bash -c conda -n kluster_test pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.kluster]: exit code: 1

See the log

(base) PS C:\Users\monoc\Downloads\kluster-kluster_0_8_9> docker build -t kluster089 . [+] Building 1.4s (19/20) => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.89kB 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu:18.04 0.8s => [ 1/17] FROM docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad 0.0s => CACHED [ 2/17] RUN apt-get update 0.0s => CACHED [ 3/17] RUN apt-get install -y git 0.0s => CACHED [ 4/17] RUN apt-get install -y wget 0.0s => CACHED [ 5/17] RUN apt install libgl1-mesa-glx -y 0.0s => CACHED [ 6/17] RUN apt-get install ffmpeg libsm6 libxext6 -y 0.0s => CACHED [ 7/17] RUN adduser --disabled-password --gecos "Non-root user" --uid 1000 --gid 100 --home /ho 0.0s => CACHED [ 8/17] RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Linux-x86_64.sh -O ~/minico 0.0s => CACHED [ 9/17] RUN echo ". /home/eyou102/miniconda3/etc/profile.d/conda.sh" >> ~/.profile 0.0s => CACHED [10/17] RUN conda init bash 0.0s => CACHED [11/17] RUN mkdir /home/eyou102/kluster 0.0s => CACHED [12/17] WORKDIR /home/eyou102/kluster 0.0s => CACHED [13/17] RUN conda update --name base --channel defaults conda 0.0s => CACHED [14/17] RUN conda create -n kluster_test python=3.8.12 0.0s => CACHED [15/17] RUN conda install -c conda-forge qgis=3.18.3 vispy=0.9.4 pyside2=5.13.2 gdal=3.3.1 h5py python-geohash 0.0s => ERROR [16/17] RUN conda -n kluster_test pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.k 0.6s

[16/17] RUN conda -n kluster_test pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.kluster: https://github.com/noaa-ocs-hydrography/kluster/issues/19 0.540 ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['/bin/bash', '-c', 'conda -n kluster_test pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.kluster']' command failed. (See above for error) https://github.com/noaa-ocs-hydrography/kluster/issues/19 0.540 https://github.com/noaa-ocs-hydrography/kluster/issues/19 0.540 CommandNotFoundError: No command 'conda kluster_test'. https://github.com/noaa-ocs-hydrography/kluster/issues/19 0.540 https://github.com/noaa-ocs-hydrography/kluster/issues/19 0.540

executor failed running [conda run -n kluster_test /bin/bash -c conda -n kluster_test pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.kluster]: exit code: 1

[ericgyounkin]

The last commands I had commented were wrong. It was before I figured out how SHELL works.

I've updated the file, can you download and retry?

https://github.com/noaa-ocs-hydrography/kluster/blob/kluster_0_8_9/dockerfile

[monocilindro]

It still doesn´t work :

executor failed running [conda run -n kluster_test /bin/bash -c python -m HSTB.kluster]: exit code: 1 .........

(base) PS C:\Users\monoc\Downloads> docker build -t kluster089 .

[+] Building 60.4s (21/21) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 1.91kB 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/ubuntu:18.04 1.5s => [auth] library/ubuntu:pull token for registry-1.docker.io 0.0s => [ 1/17] FROM docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad 0.0s => CACHED [ 2/17] RUN apt-get update 0.0s => CACHED [ 3/17] RUN apt-get install -y git 0.0s => CACHED [ 4/17] RUN apt-get install -y wget 0.0s => CACHED [ 5/17] RUN apt install libgl1-mesa-glx -y 0.0s => CACHED [ 6/17] RUN apt-get install ffmpeg libsm6 libxext6 -y 0.0s => CACHED [ 7/17] RUN adduser --disabled-password --gecos "Non-root user" --uid 1000 --gid 100 --home /ho 0.0s => CACHED [ 8/17] RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Linux-x86_64.sh -O ~/minico 0.0s => CACHED [ 9/17] RUN echo ". /home/eyou102/miniconda3/etc/profile.d/conda.sh" >> ~/.profile 0.0s => CACHED [10/17] RUN conda init bash 0.0s => CACHED [11/17] RUN mkdir /home/eyou102/kluster 0.0s => CACHED [12/17] WORKDIR /home/eyou102/kluster 0.0s => CACHED [13/17] RUN conda update --name base --channel defaults conda 0.0s => CACHED [14/17] RUN conda create -n kluster_test python=3.8.12 0.0s => CACHED [15/17] RUN conda install -c conda-forge qgis=3.18.3 vispy=0.9.4 pyside2=5.13.2 gdal=3.3.1 h5py python-geohash 0.0s => [16/17] RUN pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.kluster 56.7s => ERROR [17/17] RUN python -m HSTB.kluster 2.1s

[17/17] RUN python -m HSTB.kluster:

21 2.103 ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['/bin/bash', '-c', 'python -m HSTB.kluster']' command failed. (See above for error)

21 2.103 Traceback (most recent call last):

21 2.103 File "/home/eyou102/miniconda3/envs/kluster_test/lib/python3.8/runpy.py", line 194, in _run_module_as_main

21 2.103 return _run_code(code, main_globals, None,

21 2.103 File "/home/eyou102/miniconda3/envs/kluster_test/lib/python3.8/runpy.py", line 87, in _run_code

21 2.103 exec(code, run_globals)

21 2.103 File "/home/eyou102/miniconda3/envs/kluster_test/lib/python3.8/site-packages/HSTB/kluster/main.py", line 7, in

21 2.103 from HSTB.kluster.gui import kluster_main

21 2.103 File "/home/eyou102/miniconda3/envs/kluster_test/lib/python3.8/site-packages/HSTB/kluster/gui/kluster_main.py", line 23, in

21 2.103 import matplotlib.pyplot as plt

21 2.103 File "/home/eyou102/miniconda3/envs/kluster_test/lib/python3.8/site-packages/matplotlib/pyplot.py", line 2230, in

21 2.103 switch_backend(rcParams["backend"])

21 2.103 File "/home/eyou102/miniconda3/envs/kluster_test/lib/python3.8/site-packages/matplotlib/pyplot.py", line 275, in switch_backend

21 2.103 raise ImportError(

21 2.103 ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt' interactive framework, as 'headless' is currently running

21 2.103

---

executor failed running [conda run -n kluster_test /bin/bash -c python -m HSTB.kluster]: exit code: 1

[ericgyounkin]

I believe this is actually sort of working. The last command in the dockerfile is 'python -m HSTB.kluster' which starts the GUI. That fails in 'headless' mode as it has determined that you don't have any graphics capability. This is as far as I have gotten with Docker. If there is some way to run the GUI in Docker, I am not aware of it.

I probably should have commented that last line. But I think you can at least run the processing with the container as is, outside of the GUI.

[monocilindro]

Not sure that I got the docker image right, Once, within the docker container command line, I cannot import any library. The same if using an IDE. Besides this , even though we couldn´t get a GUI, it would be nice to run scripts in Jupyter notebooks.

[ericgyounkin]

@monocilindro I believe this might be do to the environment not being activated. I've committed a change to the dockerfile (not sure which version you have tried) and I'll show you how I use it at least. I'll add this to the docs as well.

Building the image

C:\Pydro21_Dev\NOAA\site-packages\Python38\git_repos\hstb_kluster>docker build -t kluster/ubuntu .
[+] Building 83.9s (21/21) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 1.96kB                                                                             0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:18.04                                                    0.8s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                      0.0s
 => [ 1/16] FROM docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f  0.0s
 => CACHED [ 2/16] RUN apt-get update                                                                              0.0s
 => CACHED [ 3/16] RUN apt-get install -y git                                                                      0.0s
 => CACHED [ 4/16] RUN apt-get install -y wget                                                                     0.0s
 => CACHED [ 5/16] RUN apt install libgl1-mesa-glx -y                                                              0.0s
 => CACHED [ 6/16] RUN apt-get install ffmpeg libsm6 libxext6  -y                                                  0.0s
 => CACHED [ 7/16] RUN adduser --disabled-password     --gecos "Non-root user"     --uid 1000     --gid 100     -  0.0s
 => CACHED [ 8/16] RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Linux-x86_64.sh -O  0.0s
 => CACHED [ 9/16] RUN echo ". /home/eyou102/miniconda3/etc/profile.d/conda.sh" >> ~/.profile                      0.0s
 => CACHED [10/16] RUN conda init bash                                                                             0.0s
 => CACHED [11/16] RUN mkdir /home/eyou102/kluster                                                                 0.0s
 => CACHED [12/16] WORKDIR /home/eyou102/kluster                                                                   0.0s
 => CACHED [13/16] RUN conda update --name base --channel defaults conda                                           0.0s
 => CACHED [14/16] RUN conda create -n kluster_test python=3.8.12                                                  0.0s
 => CACHED [15/16] RUN conda install -c conda-forge qgis=3.18.3 vispy=0.9.4 pyside2=5.13.2 gdal=3.3.1 h5py python  0.0s
 => [16/16] RUN pip install git+https://github.com/noaa-ocs-hydrography/kluster.git#egg=hstb.kluster              78.7s
 => exporting to image                                                                                             4.4s
 => => exporting layers                                                                                            4.3s
 => => writing image sha256:1ba82f308b0534c43e22952b28e371f5cc85965274be77de98df019a16a2de20                       0.0s
 => => naming to docker.io/kluster/ubuntu                                                                          0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

Running in interactive mode

C:\Pydro21_Dev\NOAA\site-packages\Python38\git_repos\hstb_kluster>docker run -it kluster/ubuntu
(base) eyou102@faaec62a4c1c:~/kluster$

Importing

(base) eyou102@faaec62a4c1c:~/kluster$ conda deactivate

eyou102@faaec62a4c1c:~/kluster$ conda activate kluster_test

(kluster_test) eyou102@faaec62a4c1c:~/kluster$ python

Python 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from HSTB.kluster.fqpr_convenience import *

>>> print(perform_all_processing.__doc__)

    Use fqpr_generation to process multibeam data on the local cluster and generate a sound velocity corrected,
    georeferenced xyz with uncertainty in csv files in the provided output folder.

    fqpr = fully qualified ping record, the term for the datastore in kluster

    Parameters
    ----------
    filname
        either a list of .all file paths, a single .all file path or a path to a directory with .all files
    navfiles
        list of postprocessed navigation file paths.  If provided, expects either a log file or
        weekstart_year/weekstart_week/override_datum arguments, see import_navigation
    outfold
        full file path to a directory you want to contain all the zarr folders.  Will create this folder
        if it does not exist.
    coord_system
        a valid datum identifier that pyproj CRS will accept
    vert_ref
        the vertical reference point, one of ['ellipse', 'waterline', 'NOAA MLLW', 'NOAA MHW']
    orientation_initial_interpolation
        see process_multibeam
    add_cast_files
        see process_multibeam
    skip_dask
        if True, will not start/find the dask client.  Useful for small datasets where parallel processing actually
        makes the process slower
    show_progress
        If true, uses dask.distributed.progress.
    parallel_write
        if True, will write in parallel to disk, Disable for permissions issues troubleshooting.
    vdatum_directory
        if 'NOAA MLLW' 'NOAA MHW' is the vertical reference, a path to the vdatum directory is required here

    Returns
    -------
    Fqpr
        Fqpr object containing processed data

[monocilindro]

Maybe the GUI could work using the docker image by following the guide made by the MBsystem developing team for running their software in Windows 11. https://github.com/dwcaress/MB-System/blob/master/docker/user/README-win11.md

I have adapted the launcher (sh file) accordingly but still is not loading the main menu

[ericgyounkin]

@monocilindro based on my googling, it appears this is related to not having a graphics driver in the dockerimage. See here. We could alter the dockerfile to install during the build. I believe you could also try installing after starting the docker container.

I can look at this after I finish my current task.

[epifanio]

Hi, first of all, thank you for this piece of software! It was missing!

I got the main UI working with the following docker file:

FROM nvidia/cuda:11.7.1-devel-ubuntu20.04
LABEL maintainer="massimods@met.no"

ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1

RUN apt-get update && apt-get -y dist-upgrade

RUN apt-get install -y software-properties-common
RUN apt-get update && apt-get -y dist-upgrade
RUN add-apt-repository ppa:ubuntugis/ppa
RUN apt-get update && apt-get -y dist-upgrade

RUN apt install -y mesa-utils xserver-xorg-video-all nvidia-common kmod x11-apps wget

# download from NVIDIA or install from a debian repository
# COPY NVIDIA-Linux-x86_64-515.43.04.run .
RUN wget https://us.download.nvidia.com/XFree86/Linux-x86_64/515.43.04/NVIDIA-Linux-x86_64-515.43.04.run

RUN /bin/sh NVIDIA-Linux-x86_64-515.43.04.run -a -s --no-kernel-module --install-libglvnd || \
    cat /var/log/nvidia-installer.log

RUN apt-get install  -y qgis \ 
    libqgis-dev \
    python3-qgis \
    python3-vispy \
    python3-qtpy \
    python3-pyside2.* \
    libgdal-dev \
    python3-pip \
    build-essential \
    gfortran \
    cmake \
    git \
    nano \
    python3-h5py

RUN pip install geohash

RUN python3 --version

RUN pip install git+https://github.com/epifanio/kluster.git#egg=hstb.kluster

I am using Nvidia-docker as I needed some GPU processing and I am installing the exact same video driver as the one installed on the host machine. The same dockerfile should work also starting from a standard ubuntu-20.04 image and installing the video driver directly from the debian repository (as long as they are the same drivers available on the host machine. (this should work at least for a linux host, but probably fine also for a docker running under WSL2 and win 11)

I run the container via docker compose with :

version: '3.3'
services:
    gui:
        image: epinux/kluster:latest
        runtime: nvidia
        environment:
            - NVIDIA_VISIBLE_DEVICES=all
            - DISPLAY=$DISPLAY
        command: /usr/bin/python3 -m HSTB.kluster

        volumes:
            - /tmp/.X11-unix:/tmp/.X11-unix
            - ~/.Xauthority:/root/.Xauthority
        network_mode: "host"

before running docker-compose run: xhost +local: It requires the xhost utility installed on the host machine.

[ericgyounkin]

This is great Massimo! Would you be comfortable with committing this new dockerfile, or updating the existing one?

[epifanio]

Thanks @ericgyounkin - I will clean up my fork and make a PR from a dedicated branch.