search

darktable-org / darktable

darktable is an open source photography workflow application and raw developer
https://www.darktable.org
GNU General Public License v3.0
9.47k stars 1.12k forks source link

opencl not working with nouveau/mesa/rusticl driver #16564

Open blitzgneisserin opened 4 months ago

blitzgneisserin commented 4 months ago

Describe the bug

I am trying to make darktable opencl work with the nouveau driver on Arch Linux. I have installed opencl-rusticl-mesa and set the environment variable RUSTICL_ENABLE=nouveau. I also activated rusticl in darktable opencl settings. Now I get this when I type darktable-cltest:

[anna@archlinux ~]$ darktable-cltest
darktable 4.6.1
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.2.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> ENABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     0.0113 [dt_get_sysresource_level] switched to 1 as `default'
     0.0113   total mem:       15915MB
     0.0113   mipmap cache:    1989MB
     0.0113   available mem:   7957MB
     0.0113   singlebuff:      124MB
     0.0151 [opencl_init] opencl disabled via darktable preferences
     0.0152 [opencl_init] opencl library 'libOpenCL' found on your system and loaded, preference 'default path'
     0.1253 [opencl_init] found 1 platform
[opencl_init] found 1 device

[dt_opencl_device_init]
   DEVICE:                   0: 'NV168'
   PLATFORM, VENDOR & ID:    rusticl, Mesa/X.org, ID=4318
   CANONICAL NAME:           rusticlnv168
   DRIVER VERSION:           24.0.4-arch1.2
   DEVICE VERSION:           OpenCL 3.0 , SM_20 SUPPORT
   DEVICE_TYPE:              GPU, dedicated mem
   GLOBAL MEM SIZE:          6134 MB
   MAX MEM ALLOC:            2048 MB
   MAX IMAGE SIZE:           16384 x 16384
   MAX WORK GROUP SIZE:      1024
   MAX WORK ITEM DIMENSIONS: 3
   MAX WORK ITEM SIZES:      [ 1024 1024 64 ]
   ASYNC PIXELPIPE:          NO
   PINNED MEMORY TRANSFER:   NO
   AVOID ATOMICS:            NO
   MICRO NAP:                250
   ROUNDUP WIDTH & HEIGHT    16x16
   CHECK EVENT HANDLES:      128
   TILING ADVANTAGE:         0.000
   DEFAULT DEVICE:           NO
   KERNEL BUILD DIRECTORY:   /usr/share/darktable/kernels
   KERNEL DIRECTORY:         /home/anna/.cache/darktable/cached_v3_kernels_for_rusticlNV168_2404arch12
   CL COMPILER OPTION:       
   CL COMPILER COMMAND:      -w   -DNVIDIA_SM_20=1 -DNVIDIA=1 -I"/usr/share/darktable/kernels"
/usr/include/c++/13.2.1/bits/stl_deque.h:1385: std::deque<_Tp, _Alloc>::const_reference std::deque<_Tp, _Alloc>::operator[](size_type) const [with _Tp = nv50_ir::ValueDef; _Alloc = std::allocator<nv50_ir::ValueDef>; const_reference = const nv50_ir::ValueDef&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
Aborted (core dumped)

On Ubuntu 23.10, the output is this:


anna@anna-ms7c96:~$ darktable-cltest
     0,0143 [dt_get_sysresource_level] switched to 1 as `default'
     0,0143   total mem:       15903MB
     0,0143   mipmap cache:    1987MB
     0,0143   available mem:   7951MB
     0,0143   singlebuff:      124MB
     0,0143   OpenCL tune mem: OFF
     0,0143   OpenCL pinned:   OFF
[opencl_init] opencl related configuration options:
[opencl_init] opencl: ON
[opencl_init] opencl_scheduling_profile: 'default'
[opencl_init] opencl_library: 'default path'
[opencl_init] opencl_device_priority: '*/!0,*/*/*/!0,*'
[opencl_init] opencl_mandatory_timeout: 400
     0.0168 [dt_dlopencl_init] could not find default opencl runtime library 'libOpenCL'
     0.0169 [dt_dlopencl_init] could not find default opencl runtime library 'libOpenCL.so'
[opencl_init] opencl library 'libOpenCL.so.1' found on your system and loaded
[opencl_init] found 2 platforms
[opencl_init] no devices found for Mesa (vendor) - Clover (name)
[opencl_init] found 1 device

[dt_opencl_device_init]
   DEVICE:                   0: 'NV168', NEW
   PLATFORM NAME & VENDOR:   rusticl, Mesa/X.org
   CANONICAL NAME:           rusticlnv168
   DRIVER VERSION:           23.2.1-1ubuntu3.1
   DEVICE VERSION:           OpenCL 3.0 , SM_20 SUPPORT
   DEVICE_TYPE:              GPU
   GLOBAL MEM SIZE:          6134 MB
   MAX MEM ALLOC:            2048 MB
   MAX IMAGE SIZE:           16384 x 16384
   MAX WORK GROUP SIZE:      1024
   MAX WORK ITEM DIMENSIONS: 3
   MAX WORK ITEM SIZES:      [ 1024 1024 64 ]
   ASYNC PIXELPIPE:          NO
   PINNED MEMORY TRANSFER:   NO
   MEMORY TUNING:            NO
   FORCED HEADROOM:          400
   AVOID ATOMICS:            NO
   MICRO NAP:                250
   ROUNDUP WIDTH:            16
   ROUNDUP HEIGHT:           16
   CHECK EVENT HANDLES:      128
   TILING ADVANTAGE:         0.000
   DEFAULT DEVICE:           NO
   KERNEL BUILD DIRECTORY:   /usr/share/darktable/kernels
   KERNEL DIRECTORY:         /home/anna/.cache/darktable/cached_v1_kernels_for_rusticlNV168_23211ubuntu31
   CL COMPILER OPTION:       -cl-fast-relaxed-math
     1.5749 [opencl_build_program] could not build program: CL_BUILD_PROGRAM_FAILURE
     1.5749 [dt_opencl_device_init] failed to compile program `bilateral.cl'!
[opencl_init] no suitable devices found.
[opencl_init] FINALLY: opencl is NOT AVAILABLE and NOT ENABLED.

I guess the problem is the fact that -cl-fast-relaxed-math is missing on Arch. How can it be added?

GPU Hardware is Nvidia GTX 1660 Super

Steps to reproduce

see above

Expected behavior

opencl should work

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you obtain darktable from?

distro packaging

darktable version

4.6.1

What OS are you using?

Linux

What is the version of your OS?

rolling

Describe your system?

see description

Are you using OpenCL GPU in darktable?

I dont know

If yes, what is the GPU card and driver?

Nvidia GTX 1660 Super + nouveau/mesa/rusticl

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

jenshannoschwalm commented 4 months ago

@karolherbst ? May i ping you here ...

karolherbst commented 4 months ago

yeah... so that's a mesa bug and the nouveau driver's compiler is in a sorry state atm. It's better to try out zink and nvk here, but that should wait for mesa 24.1 and I haven't check if that works.

jenshannoschwalm commented 4 months ago

yeah... so that's a mesa bug and the nouveau driver's compiler is in a sorry state atm.

Currently dt compiles with -DNVIDIA_SM_20=1 -DNVIDIA=1 when detecting ID=4318. I think that might be wrong? Can you comment please @karolherbst ?

karolherbst commented 4 months ago

yeah... so that's a mesa bug and the nouveau driver's compiler is in a sorry state atm.

Currently dt compiles with -DNVIDIA_SM_20=1 -DNVIDIA=1 when detecting ID=4318. I think that might be wrong? Can you comment please @karolherbst ?

what's ID=4318? the PCI device id? But yeah, the NV168/TU118 is SM75, but not sure what SM levels you care about here.

blitzgneisserin commented 4 months ago

How can zink and nvk be used for darktable/opencl?

karolherbst commented 4 months ago

How can zink and nvk be used for darktable/opencl?

through rusticl. It should work but I don't know if nvk implements all the required extensions yet. It's something on my todo list to figure out, but I think it works. Just do RUSTICL_ENABLE=zink instead. Might also have to point to a specific vulkan impl if you have multiple installed.

blitzgneisserin commented 4 months ago

do RUSTICL_ENABLE=zink instead

Ok. Now I installed vulkan-nouveau-git and set RUSTICL_ENABLE=zink and I get this (opencl available and NOT enabled):


[anna@archlinux ~]$ darktable-cltest
darktable 4.6.1
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.2.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> ENABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     0.0113 [dt_get_sysresource_level] switched to 1 as `default'
     0.0113   total mem:       15915MB
     0.0113   mipmap cache:    1989MB
     0.0113   available mem:   7957MB
     0.0113   singlebuff:      124MB
     0.0152 [opencl_init] opencl disabled via darktable preferences
     0.0153 [opencl_init] opencl library 'libOpenCL' found on your system and loaded, preference 'default path'
     0.4250 [opencl_init] found 2 platforms
     0.4250 [opencl_init] no devices found for Mesa (vendor) - Clover (name)
[opencl_init] found 1 device

[dt_opencl_device_init]
   DEVICE:                   0: 'zink Vulkan 1.3(NVIDIA GeForce GTX 1660 SUPER (NVK TU116) (MESA_NVK))'
   PLATFORM, VENDOR & ID:    rusticl, Mesa/X.org, ID=4318
   CANONICAL NAME:           rusticlzinkvulkan13
   DRIVER VERSION:           24.0.4-arch1.2
   DEVICE VERSION:           OpenCL 3.0 
   DEVICE_TYPE:              GPU, dedicated mem
   GLOBAL MEM SIZE:          6400 MB
   MAX MEM ALLOC:            2048 MB
   MAX IMAGE SIZE:           32768 x 32768
   MAX WORK GROUP SIZE:      1024
   MAX WORK ITEM DIMENSIONS: 3
   MAX WORK ITEM SIZES:      [ 1024 1024 64 ]
   ASYNC PIXELPIPE:          NO
   PINNED MEMORY TRANSFER:   NO
   AVOID ATOMICS:            NO
   MICRO NAP:                250
   ROUNDUP WIDTH & HEIGHT    16x16
   CHECK EVENT HANDLES:      128
   TILING ADVANTAGE:         0.000
   DEFAULT DEVICE:           NO
   KERNEL BUILD DIRECTORY:   /usr/share/darktable/kernels
   KERNEL DIRECTORY:         /home/anna/.cache/darktable/cached_v3_kernels_for_rusticlzinkVulkan13_2404arch12
   CL COMPILER OPTION:       
   CL COMPILER COMMAND:      -w   -DNVIDIA=1 -I"/usr/share/darktable/kernels"
   KERNEL LOADING TIME:       0.1882 sec
[opencl_init] OpenCL successfully initialized. internal numbers and names of available devices:
[opencl_init]           0       'rusticl zink Vulkan 1.3'
     0.6145 [opencl_init] FINALLY: opencl is AVAILABLE and NOT ENABLED.
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       -1      0       0       -1
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       0       0       0       0
[opencl_synchronization_timeout] synchronization timeout set to 200
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       -1      0       0       -1
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       0       0       0       0
[opencl_synchronization_timeout] synchronization timeout set to 200
karolherbst commented 4 months ago

do RUSTICL_ENABLE=zink instead

Ok. Now I installed vulkan-nouveau-git and set RUSTICL_ENABLE=zink and I get this (opencl available and NOT enabled):


[anna@archlinux ~]$ darktable-cltest
darktable 4.6.1
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.2.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> ENABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     0.0113 [dt_get_sysresource_level] switched to 1 as `default'
     0.0113   total mem:       15915MB
     0.0113   mipmap cache:    1989MB
     0.0113   available mem:   7957MB
     0.0113   singlebuff:      124MB
     0.0152 [opencl_init] opencl disabled via darktable preferences
     0.0153 [opencl_init] opencl library 'libOpenCL' found on your system and loaded, preference 'default path'
     0.4250 [opencl_init] found 2 platforms
     0.4250 [opencl_init] no devices found for Mesa (vendor) - Clover (name)
[opencl_init] found 1 device

[dt_opencl_device_init]
   DEVICE:                   0: 'zink Vulkan 1.3(NVIDIA GeForce GTX 1660 SUPER (NVK TU116) (MESA_NVK))'
   PLATFORM, VENDOR & ID:    rusticl, Mesa/X.org, ID=4318
   CANONICAL NAME:           rusticlzinkvulkan13
   DRIVER VERSION:           24.0.4-arch1.2
   DEVICE VERSION:           OpenCL 3.0 
   DEVICE_TYPE:              GPU, dedicated mem
   GLOBAL MEM SIZE:          6400 MB
   MAX MEM ALLOC:            2048 MB
   MAX IMAGE SIZE:           32768 x 32768
   MAX WORK GROUP SIZE:      1024
   MAX WORK ITEM DIMENSIONS: 3
   MAX WORK ITEM SIZES:      [ 1024 1024 64 ]
   ASYNC PIXELPIPE:          NO
   PINNED MEMORY TRANSFER:   NO
   AVOID ATOMICS:            NO
   MICRO NAP:                250
   ROUNDUP WIDTH & HEIGHT    16x16
   CHECK EVENT HANDLES:      128
   TILING ADVANTAGE:         0.000
   DEFAULT DEVICE:           NO
   KERNEL BUILD DIRECTORY:   /usr/share/darktable/kernels
   KERNEL DIRECTORY:         /home/anna/.cache/darktable/cached_v3_kernels_for_rusticlzinkVulkan13_2404arch12
   CL COMPILER OPTION:       
   CL COMPILER COMMAND:      -w   -DNVIDIA=1 -I"/usr/share/darktable/kernels"
   KERNEL LOADING TIME:       0.1882 sec
[opencl_init] OpenCL successfully initialized. internal numbers and names of available devices:
[opencl_init]           0       'rusticl zink Vulkan 1.3'
     0.6145 [opencl_init] FINALLY: opencl is AVAILABLE and NOT ENABLED.
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       -1      0       0       -1
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       0       0       0       0
[opencl_synchronization_timeout] synchronization timeout set to 200
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       -1      0       0       -1
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities]           image   preview export  thumbs  preview2
[dt_opencl_update_priorities]           0       0       0       0       0
[opencl_synchronization_timeout] synchronization timeout set to 200

I think you just need to enable it again or soething? but yeah, please let me know how that works, though you might run into bugs. In any case, not much we can do here as bug reports against 24.0 are futile as nvk is still experimental. So you should wait until 24.1 in case you run into any issues and file bugs against mesa there.

blitzgneisserin commented 4 months ago

@karolherbst yea, it kind does work. Turns out, I just had to start darktable from the very same terminal window where I set the rusticl environment variable. However, it seems to crash often and sometimes the preview is strange. When it crashes, I get outputs like this in the terminal:

[anna@archlinux ~]$ darktable
SPIR-V parsing FAILED:
    In file ../mesa/src/compiler/spirv/spirv_to_nir.c:369
    Type mismatch for SPIR-V value %622
    23268 bytes into the SPIR-V binary
Trace/breakpoint trap (core dumped)

Well and it seems that it works better on X11, apparently the strange behavior is more frequent on Wayland/Xwayland.

Performance-wise it's comparable to the proprietary Nvidia driver, though I didn't measure with darktable -d perf.

Edit: as far as the preview is concerned, a frequent issue is the fact that it just turns black when I move a slider or switch on a module. If I then change the zoom level or hide/unhide a panel, it comes back. Pretty annoying. Well in theory it works but not really usable...

karolherbst commented 4 months ago

@karolherbst yea, it kind does work. Turns out, I just had to start darktable from the very same terminal window where I set the rusticl environment variable. However, it seems to crash often and sometimes the preview is strange. When it crashes, I get outputs like this in the terminal:

[anna@archlinux ~]$ darktable
SPIR-V parsing FAILED:
    In file ../mesa/src/compiler/spirv/spirv_to_nir.c:369
    Type mismatch for SPIR-V value %622
    23268 bytes into the SPIR-V binary
Trace/breakpoint trap (core dumped)

Well and it seems that it works better on X11, apparently the strange behavior is more frequent on Wayland/Xwayland.

Performance-wise it's comparable to the proprietary Nvidia driver, though I didn't measure with darktable -d perf.

Edit: as far as the preview is concerned, a frequent issue is the fact that it just turns black when I move a slider or switch on a module. If I then change the zoom level or hide/unhide a panel, it comes back. Pretty annoying. Well in theory it works but not really usable...

yeah.. I doubt it would work on any version before 24.1 as zink is still kinda experimental with rusticl. But anyway, you should probably file bugs against mesa running into any issues with rusticl.

github-actions[bot] commented 2 months ago

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

jenshannoschwalm commented 3 weeks ago

@blitzgneisserin anything new worthwhile to mention?

blitzgneisserin commented 3 weeks ago

@jenshannoschwalm re tested about 2 or 3 weeks ago. Seems there is progress. My impression was that the strange behaviour/previews are gone, but there are still quite a few crashes when newer scene referred modules are used, such as diffuse/sharpen or color balance rgb.

jenshannoschwalm commented 3 weeks ago

This is on master or 4.8.1 ?

Maybe worth to have a fresh -d pipe -d opencl' log but no haste ...

blitzgneisserin commented 3 weeks ago

@jenshannoschwalm I guess the version that's in the arch repo, so probably 4.8 or 4.8.1.

jenshannoschwalm commented 3 weeks ago

As you seem to be testing rusticl from time to time i would be glad if you could report if you find it to be stable, ok?

blitzgneisserin commented 3 weeks ago

@jenshannoschwalm ok. Don't forget to remind me though. Rightnow I have some newphotos and I guess I could test again. But in general a lot of things are going on right now in my life so I might need some reminders.

jenshannoschwalm commented 3 weeks ago

Things going right is always good. All the best to you!

blitzgneisserin commented 3 weeks ago

@jenshannoschwalm I just tried to use the Appimage dev build with Rusticl and I get this:

╭─anna@anna in ~/Downloads as 🧙 took 16s
 ╰─λ ./Darktable-4.9.0.AppImage
Gtk-Message: 08:31:59.559: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 08:31:59.560: Failed to load module "colorreload-gtk-module"
MESA: error: ZINK: vkCreateInstance failed (VK_ERROR_INCOMPATIBLE_DRIVER)

Would it make sense to compile dt 4.9?

karolherbst commented 3 weeks ago

you might have to set VK_ICD_FILENAMES and point it to wherever nouveau_icd.x86_64.json is installed (e.g. VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nouveau_icd.x86_64.json. Device selection with zink atm is quite janky and I'll need to rework it.

Also this all really depends on what version of mesa is getting used, and for nouveau/nvk specifically it's usually best to try with the latest as it's still in heavy development. Best to use 24.2 or newer.

I've done some CL CTS testing recently and there still a few bugs to figure out and darktable seems to work fine with recent mesa main on my Turing GPU, but random filters can still be buggy I guess.

blitzgneisserin commented 3 weeks ago

@karolherbst I think this is an issue with dt 4.9 (dev), since 4.8.x can use zink/vulkan, at least many modules.

jenshannoschwalm commented 3 weeks ago

Thanks for reporting. Can you comment on amd status with rusticl?

blitzgneisserin commented 3 weeks ago

@jenshannoschwalm Do I need an amd gpu for that? Because I don't have one. I think I need precise instructions for that.

jenshannoschwalm commented 3 weeks ago

Ok, I thought you have one.

I would love to make rusticl the default for AMD in darktable if it is stable...