search

clEsperanto / pyclesperanto_prototype

GPU-accelerated bio-image analysis focusing on 3D+t microscopy image data
http://clesperanto.net
BSD 3-Clause "New" or "Revised" License
208 stars 44 forks source link

POCL BUILD_PROGRAM_FAILURE #332

Closed MariusCausemann closed 5 months ago

MariusCausemann commented 5 months ago

Hi, I'm facing an issue using a conda installed pocl for opencl. E.g. running:

import pyclesperanto_prototype as cle
cle.select_device("NVIDIA")
#<NVIDIA A100-SXM4-80GB on Platform: Portable Computing Language (2 refs)>

from skimage import data
import numpy as np
import matplotlib.pyplot as plt

from skimage.filters import sobel
from skimage.measure import label
from skimage.segmentation import watershed, expand_labels
from skimage.color import label2rgb
from skimage import data

coins = data.coins()

# Make segmentation using edge-detection and watershed.
edges = sobel(coins)

# Identify some background and foreground pixels from the intensity values.
# These pixels are used as seeds for watershed.
markers = np.zeros_like(coins)
foreground, background = 1, 2
markers[coins < 30.0] = background
markers[coins > 150.0] = foreground

ws = watershed(edges, markers)
seg1 = label(ws == foreground)

cle.dilate_labels(seg1)

gives:

9 warnings generated.
Traceback (most recent call last):
  File "/home/mariusca/emimesh/pocltest.py", line 30, in <module>
    cle.dilate_labels(seg1)
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_plugin_function.py", line 71, in worker_function
    return function(*bound.args, **bound.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier4/_dilate_labels.py", line 45, in dilate_labels
    copy(labeling_source, flip)
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_plugin_function.py", line 71, in worker_function
    return function(*bound.args, **bound.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier1/_copy.py", line 36, in copy
    execute(__file__, '../clij-opencl-kernels/kernels/copy_' + str(len(destination.shape)) + 'd_x.cl', 'copy_' + str(len(destination.shape)) + 'd', destination.shape, parameters)
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_execute.py", line 35, in execute
    return Backend.get_instance().get().execute(anchor, opencl_kernel_filename, kernel_name, global_size, parameters, prog, constants, image_size_independent_kernel_compilation, device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_opencl_backend.py", line 41, in execute
    return execute(anchor, opencl_kernel_filename, kernel_name, global_size, parameters, prog, constants, image_size_independent_kernel_compilation, device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_opencl_execute.py", line 311, in execute
    prog = device.program_from_source("\n".join(defines))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_device.py", line 28, in program_from_source
    return OCLProgram(src_str=source, dev=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyclesperanto_prototype/_tier0/_program.py", line 28, in __init__
    self.build(options=build_options)
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyopencl/__init__.py", line 535, in build
    self._prg, was_cached = self._build_and_catch_errors(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyopencl/__init__.py", line 583, in _build_and_catch_errors
    raise err
pyopencl._cl.RuntimeError: clBuildProgram failed: BUILD_PROGRAM_FAILURE - clBuildProgram failed: BUILD_PROGRAM_FAILURE - clBuildProgram failed: BUILD_PROGRAM_FAILURE

Build on <pyopencl.Device 'NVIDIA A100-SXM4-80GB' on 'Portable Computing Language' at 0x55555681e820>:

warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:25:26: unsupported OpenCL extension 'cl_khr_3d_image_writes' - ignoring
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:27:26: OpenCL extension 'cl_amd_printf' unknown or does not require pragma - ignoring
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:573:17: implicit conversion from 'long' to 'float' changes value from 4294967295 to 4294967296
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:583:17: implicit conversion from 'int' to 'float' changes value from 2147483647 to 2147483648
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:594:17: implicit conversion from 'long long' to 'float' changes value from 18446744073709551615 to 18446744073709551616
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:595:16: implicit conversion from 'long long' to 'uint' (aka 'unsigned int') changes value from 18446744073709551615 to 4294967295
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:604:17: implicit conversion from 'long' to 'float' changes value from 9223372036854775807 to 9223372036854775808
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:605:16: implicit conversion from 'long' to 'int' changes value from 9223372036854775807 to -1
warning: /home/mariusca/.cache/pocl/kcache/tempfile_zzMAJF.cl:608:16: implicit conversion from 'long long' to 'int' changes value from -9223372036854775808 to 0

(options: -I /home/mariusca/miniconda3/envs/opencl/lib/python3.12/site-packages/pyopencl/cl)
(source saved as /tmp/tmpeh7djqmg.cl)
still have references to IRs - can't release LLVM context !
Aborted (core dumped)

with an environment created with: mamba create -n opencl -c conda-forge pyclesperanto-prototype pocl

# packages in environment at /home/mariusca/miniconda3/envs/opencl:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aom                       3.8.2                h59595ed_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
blosc                     1.21.5               h0f2a231_0    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-blosc2                  2.13.2               hb4ffafa_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
charls                    2.4.2                h59595ed_0    conda-forge
contourpy                 1.2.0           py312h8572e83_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hd590300_0    conda-forge
fonttools                 4.49.0          py312h98912ed_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
imagecodecs               2024.1.1        py312h55207c9_2    conda-forge
imageio                   2.34.0             pyh4b66e23_0    conda-forge
importlib-metadata        7.0.2              pyha770c72_0    conda-forge
jxrlib                    1.1                  hd590300_3    conda-forge
kiwisolver                1.4.5           py312h8572e83_1    conda-forge
lazy_loader               0.3                pyhd8ed1ab_0    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libavif16                 1.0.4                hd9d6309_2    conda-forge
libblas                   3.9.0           21_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libclang-cpp15            15.0.7          default_hb11cfb5_4    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libhwloc                  2.9.3           default_h554bfaf_1009    conda-forge
libhwy                    1.0.7                h00ab1b0_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
libjxl                    0.10.1               h5b01ea3_0    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libllvm15                 15.0.7               hb3ce162_4    conda-forge
libllvmspirv15            15.0.0               h0cdce71_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libsqlite                 3.45.2               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.6               h232c23b_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
llvm-spirv-15             15.0.0               h0cdce71_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
mako                      1.3.2              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.5           py312h98912ed_0    conda-forge
matplotlib-base           3.8.3           py312he5832f3_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
numpy                     1.26.4          py312heda63a1_0    conda-forge
ocl-icd                   2.3.2                hd590300_0    conda-forge
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openssl                   3.2.1                hd590300_1    conda-forge
packaging                 24.0               pyhd8ed1ab_0    conda-forge
pillow                    10.2.0          py312hf3581a9_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
pocl                      5.0                  h03a6ac1_2    conda-forge
pocl-core                 5.0                  hdaecddf_2    conda-forge
pocl-cpu                  5.0                  he901f76_2    conda-forge
pocl-cpu-minimal          5.0                  h5ccd973_2    conda-forge
pocl-cuda                 5.0                  hdaecddf_2    conda-forge
pocl-remote               5.0                  h5ccd973_2    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyclesperanto-prototype   0.24.2             pyhd8ed1ab_0    conda-forge
pyopencl                  2024.1          py312hfb10629_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
python                    3.12.2          hab00c5b_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pytools                   2023.1.1           pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1           py312hc7c0aa3_1    conda-forge
rav1e                     0.6.6                he8a937b_2    conda-forge
readline                  8.2                  h8228510_1    conda-forge
scikit-image              0.22.0          py312hfb8ada1_2    conda-forge
scipy                     1.12.0          py312heda63a1_2    conda-forge
setuptools                69.2.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
svt-av1                   2.0.0                h59595ed_0    conda-forge
tifffile                  2024.2.12          pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
transforms3d              0.4.1              pyhd8ed1ab_0    conda-forge
typing_extensions         4.10.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zfp                       1.0.1                h59595ed_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib-ng                   2.0.7                h0b41bf4_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

The same code runs on with a system provided opencl driver. What could be the problem here?

haesleinhuepf commented 5 months ago

Hi @MariusCausemann ,

do you think it would be possible in your scenario to avoid pocl? It's very slow and I recommend using a GPU and corresponding drivers instead.

Best, Robert

MariusCausemann commented 5 months ago

Hi Robert, I'm using a GPU and corresponding drivers for my main workflow, but it would be very useful to have a CPU version too - mainly for CI testing and reproducibility. I was hoping that I could specify one conda environment with pocl, which would run on different machines with and without a dedicated GPU. I see that pyclesperanto also uses pocl for CI?

haesleinhuepf commented 5 months ago

I see that pyclesperanto also uses pocl for CI?

Yes we do. And only for this. I guess you saw our installation config for the CI here. Maybe there is an obvious difference with your environment?

haesleinhuepf commented 5 months ago

What operating system are you running on?

MariusCausemann commented 5 months ago

I'm running on ubuntu 22.04. I found a setup with intel-opencl-icd from conda-forge for CPU support and system installed drivers for my NVIDIA GPU, so the pocl issue is not urgent for me anymore. Should I close the issue or keep it open?

haesleinhuepf commented 5 months ago

awesome, great you found a solution! I'm closing this because pocl seems out of scope for clesperanto as it would hardly accelerate processing.

Thanks for the feedback!