[Request]: Add CUDA support

obreo commented 9 months ago

Tell us the problem or your need

I'm trying to install Davinci resolve using bottles' Caffe 9.2 runner, The software is running and recognizing the Nvidia driver, however, it is recognizing OpenCL but showing 'failed to initialize OpenCL' error in the logs. I'm trying to add Cuda to the option so davinci would pick it up instead of openCL (the davinci settings show only the OpenCL available).

Based on my search, it seems Winehq supports CUDA for wine-staging only. But it looks like they removed the nvcuda.dll and cuda library in the recent winehq builds, as I cant find it in the library https://wiki.winehq.org/Wine-Staging_CUDA

So I'm trying to add that to wine installed on Bottles runner, but with no luck as I'm unable to locate the configuration path exactly.

Here is the cuda library supported for wine: https://github.com/SveSop/nvidia-libs

I hope you add this library as a feature so it can be either chosen from the dependencies or the winecfg.

Based on my debugging experience with davinci resolve, the only runner was able to make it work with nvidia is Cafe 9.2 version. I believe if Davinci was able to pick the CUDA support from wine then it will work - as the software is functioning but playback is not. Running Davinci resolve on wine can help users use h.264 and h.265 support on linux, this is why i am working on this fix. It looks promising.

I'd love to collaborate. Thanks

Describe the solution you'd like

CUDA support is missing, it is a library that can be added to wine-staging.

Additional context and references

https://wiki.winehq.org/Wine-Staging_CUDA

Here is the cuda library supported for wine: https://github.com/SveSop/nvidia-libs

orowith2os commented 9 months ago

I recommend you to get some definitely working steps to reproduce a working CUDA environment in Wine first, and then create something akin to how dxvk and latencyflex and vkd3d ship their DLLs that Bottles can pull from in order to set things up appropriately.

Alternatively, or also, get OpenCL working by way of rusticl, that way Davinci would work on AMD and Intel, plus NVK, as well.

obreo commented 8 months ago

Ok I got CUDA running on Bottles.

Method: Runner: Cafe 9.2 OS: PopOs 22.02 Architecture: 64bit (davinci does not run over 32bit) Settings: discrete gpu and other gpu settings ON. Dependecies: DirectX11/10 libraries & visualc++ redistributable libraries.

Workaround for Resolve: I needed to download directml.dll manually and insert it in System32 directory to let davinci run.

Installing CUDA: What I did was installing the dlls from the source I mentioned in the post, then copied them to the directory where the Cafe runner is installed. I then added the dll library for nvcuda.dll in the winecfg.

I had to install the CUDA-toolkit from nvidia and verify that cuda is installed on my PopOs machine.

I needed to install ffshow dependency for windows codecs.

Then it was detected by Davinci Resolve.

Result:

Davinci was able to run.
The playback was running but cutting a lot of frames and showing Media Offline for those frames. The playback sound was running fine.
I checked the davinci logs and it gives error says: Failed to decode.
I get ole error in the terminal while runnig davinci, although the ole libraries were installed in winecfg.
Generating media and proxy files didnt work and i think that is related to the previous error.

Thoughts:

I think there is GPU driver/library incompatibility/missing libraries that is causing the mentioned error and the skipping of frames.

Note: I use davinci resolve on my Windows machine using CUDA and it works smoothly.

When I choose openCL the playback in PopOs, it does not work. I am not sure if this is related to my GPU as I have never tried running davinci over OpenCL in my Windows machine. So it could be running out of the box and maybe missing.

kurageart commented 8 months ago

Hi, I also need this.. I can confirm that the cuda library from github it works fine for most of what I need (on a standard wine installation), but I haven't try to run davinci on wine .. By the way, in bottles I downloaded the wine-ge runner , stating on his page that is adding" Nvidia CUDA support for phsyx and nvapi" . But even opening cudaZ I get the "Cuda not found! Please update your Nvidia driver and try again". Having this working on bottles would save the hassle to enable multiarch ,installing wine at all, and keep everything in flatpaks

kurageart commented 8 months ago

What I did was installing the dlls from the source I mentioned in the post, then copied them to the directory where the Cafe runner is installed. I then added the dll library for nvcuda.dll in the winecfg.

Hi obreo , can You please tell witch file You copied and exactly where? I'm trying to use the "release" version of https://github.com/SveSop/nvidia-libs , since on Debian and Clearlinux it won't compile . Installing it using the regular setup_nvlibs.sh on the real wineprefix works fine, but I would be happy to stay away from multiarch etc and simply having it work on bottles ... The script itself requires a real wine installation to work, but all is doing from my modest understanding is symlinking the dll to the wine/c/windows/system32 directory ..

obreo commented 8 months ago

What I did was installing the dlls from the source I mentioned in the post, then copied them to the directory where the Cafe runner is installed. I then added the dll library for nvcuda.dll in the winecfg.

Hi obreo , can You please tell witch file You copied and exactly where? I'm trying to use the "release" version of https://github.com/SveSop/nvidia-libs , since on Debian and Clearlinux it won't compile . Installing it using the regular setup_nvlibs.sh on the real wineprefix works fine, but I would be happy to stay away from multiarch etc and simply having it work on bottles ... The script itself requires a real wine installation to work, but all is doing from my modest understanding is symlinking the dll to the wine/c/windows/system32 directory ..

Hi, I downloaded the file in https://github.com/SveSop/nvidia-lib from release, but choose the nvidia-libs-0.7.14.tar.xz , extract it, and you'll have two folders, one of them is nvida-libs-release, inside it, there are two folders, lib and lib64, copy the .dll and .so files inside each of these folders based on their architecture to the relative folders inside the cafe runner directory. Cafe runner itself is wine and located -sorry i dont remember the exact path as I'm working on Windows currently but something similar to this - in /home/USER/.var/bottles/data/bottle/runners/cafe (The one you installed by bottles)/

Inside that directory you will add the dll files inside the lib and lib64 to the relative folders in the cafe runner path, similarly for the .so . Make sure that you put the right files in the right architecture folder for x86_64 and i386 architectures.

After that, open winecfg from the bottle you created in bottles, from library tab, you should find the nvcuda.dll file, add it along with the rest of dll files you are copied above.

Finally, add the nvcuda.dll file inside the System32 directory in the c/windows/system32/ path

kurageart commented 8 months ago

Thank You! unfortunately it still don't wanna work..I followed what you said step by step. Could it be cause I'm using an amd apu as display card, and a discrete 3060 not connected to any monitor? or because i'm using a flatpak version of bottles?

SveSop commented 8 months ago

Just a quick chime in to this, as i tested it using bottles and creating a bottle and running some cuda samples.

The "tricky" part is that for wine-9 and late wine-8 versions of wine, it is no longer possible to use the "fake" .dll's that is produced from the nvidia-libs archive. This is mainly due to how newer versions of wine load libraries that "talk to" native linux libs, and tbh i have not started that work yet.

However, using the dll.so file AS a .dll (basically renaming nvcuda.dll.so -> nvcuda.dll still works and loads.. for now).

Not really familiar with using bottles and creating scripts that could be used for the GUI in a similar manner as DXVK and so on, but it might be just as good an idea as any if that would be interesting. I actually have nvcuda as a separate repo where i do the development and then just import this to nvidia-libs. So if some sort of "NVCUDA" toggle thing would be of use for the bottles GUI, i could provide "release" type of binaries on the nvcuda repo for you to download?

https://github.com/SveSop/nvcuda/tree/devel (The master branch is where wine-staging left off a couple of years ago)

EDIT: I made a change to how nvcuda is built standalone here: https://github.com/SveSop/nvcuda/tree/buildscript This would build nvcuda as nvcuda.dll and not .dll.so (even if it is an ELF library), and put it in lib32 and lib64 folder. If this could be more useful and easier to implement as a addon "NVCUDA" thing to bottles, let me know and i could provide some sort of "release" binaries for you to use if that is interesting (Suppose there are some sort of manual intervention to update available versions whenever there is a release of libs like DXVK or whatnot?)

TiZ-HugLife commented 7 months ago

Hi there. I've actually been using nvidia-libs in Bottles for a long time, because Bottles is how I run Daz Studio. The Iray renderer is pretty much a requirement for Daz Studio, and Iray requires CUDA. It currently takes a little bit of manual finagling due to the fact that the installer script wants to create a symlink to the library's actual location, so I'll outline the steps I take.

First, I create a folder: $XDG_DATA_HOME/bottles/nvidia-libs to keep all the versions in. I unpack the release archive into its own folder here. The last release was 0.7.14, so I extract the folder inside the archive as $XDG_DATA_HOME/bottles/nvidia-libs/0.7.14.

Now, if you want to use the installer script, it's tricky because you have to make the right set of components intersect:

A shell for the specific Flatpak environment for Bottles
The correct Wine prefix
The correct Wine runner bins
The correct paths for all of those, as seen by the Flatpak environment

So here's the commands I run and the terminal output:

❯ flatpak run --command=bash com.usebottles.bottles
[📦 com.usebottles.bottles ~]$ cd $XDG_DATA_HOME/bottles/nvidia-libs/0.7.14
[📦 com.usebottles.bottles 0.7.14]$ export WINEPREFIX=$(realpath ../../bottles/DAZ)
[📦 com.usebottles.bottles 0.7.14]$ export PATH=$(realpath ../../runners/caffe-8.21/bin):$PATH
[📦 com.usebottles.bottles 0.7.14]$ ./setup_nvlibs.sh install
[1/2] nvcuda :
    Creating DLL override... 
    Creating symlink to nvcuda.dll.so... 
[2/2] nvapi :
    Creating DLL override... 
    Creating symlink to nvapi.dll... 
[1/3] 64 bit nvcuda :
    Creating DLL override... 
    Creating symlink to nvcuda.dll.so... 
[2/3] 64 bit nvoptix :
    Creating DLL override... 
    Creating symlink to nvoptix.dll.so... 
[3/3] 64 bit nvapi64 :
    Creating DLL override... 
    Creating symlink to nvapi64.dll... 
Symlinks created in /home/tiz/.var/app/com.usebottles.bottles/data/bottles/bottles/DAZ. Do NOT remove this source folder!
OBS! NVML is NOT enabled. See Readme_nvml.txt for info
You need to REMOVE old overrides if older version of nvml have been used in /home/tiz/.var/app/com.usebottles.bottles/data/bottles/bottles/DAZ

I don't go through the process of enabling NVML because Iray doesn't need it. But after that's done, I can start Daz Studio and use GPU-accelerated Iray. It's just as fast as native Linux Iray Server. CUDA-Z works too, for a much more accessible way to test the setup.

Contrary to what @SveSop says--and I'm not sure why, because they would certainly know better than me--the distribution as-is works just fine with all Caffe versions from 7.2 through 9.2. But Caffe might be special in some crucial ways I'm not aware of. I actually can't run Daz Studio with any other branch, come to think of it, because it complains that it's being run as administrator.

It would definitely be better to have nvidia-libs integrated into Bottles itself if possible, because running the installer script is very clunky. But it's probably not actually necessary exactly as I'm doing it. It doesn't seem like the installer script does anything particularly magical; it makes the DLL files available in the prefix, and then sets up DLL overrides in the registry for each one. I don't really care about the DLL files being symlinks, because I don't want them to update without me explicitly doing so anyways. So I could probably just copy them and set up the DLL overrides manually.

SveSop commented 7 months ago

I am not horribly well at explaining myself..

Atm nvlibs is built with "fake" dll's in the nvidia-libs-release/lib64/wine/x86_64-windows folder, and it is kind of easy to think THAT is what you should symlink/copy into the wineprefix (its a .dll file after all). This does not work... Prior to wine-8.x it was actually possible to use this fake .dll along with the dll.so in a way so you did not need to use dll-override and such, but this no longer works.

It just MAY seem "not correct" for some to think they need to symlink a linux .so library like that, and that was kind of what i was hinting at. It works just fine like it is, it just could possibly be a bit misleading, especially when comparing it to DXVK/VKD3D that is built as native PE libs.

I understand it sounded somewhat more serious than i intended 😄

evertvorster commented 2 months ago

Hi there. Just discovered this thread, and it is remarkably similar to what I am attempting to do. The situation: I prefer to use Linux, and am a content creator. I just bought a shiny Insta360 X4. The software that is used to edit the footage is windows only. I am able to run this insta360 studio in bottles, but it does not use the GPU for acceleration of the video encoding and decoding. While it is possible to edit with just the CPU, having a little help from the GPU is desired.

Running GPUZ in the bottle shows that my video card is detected, but it shows that it is not CUDA capable. (nVidia RTX 4090 is definitely CUDA capable)

So, it would be nice to have the nvidia-libs as a selectable and installable dependency in bottles. At the moment I am fumbling through the instructions on nvidia-libs github and here. I managed to build nvidia-libs locally, but installing it into my bottle is still eluding me.

Do I have to use the caffe runner, or can I select one of the other ones? With Caffe, the nvidia card is not detected, with sys-wine it is.

evertvorster commented 2 months ago

@SveSop from a user perspective, it would be easier to have an "nvcuda" thing that you can just plug into bottles and then have cuda capabilities for that bottle. Of course, this is a lot more work for you....

SveSop commented 2 months ago

@evertvorster I am not really sure where to start to make a PR to bottles with the necessary changes to .xml files, and whatnot that would be needed. I am willing to provide a bare-bones binary for nvcuda, but unless someone know what and how to edit the bottles settings/gui/xml and scripts to do this... well, i am sorry 😢

evertvorster commented 2 months ago

Hi there. Many thanks for taking the time to reply to this. A good starting point would be that barebones thing. Once we have a provable cuda working in a bottle, then we can go from there to package it better.

SveSop commented 2 months ago

I have added a simplified release here: https://github.com/SveSop/nvcuda/releases This only has ELF renamed to .DLL, and works just fine if copied directly into the bottles prefix without the need for overrides. This could be subject to change in the case a wine runner is built WITH the old nvcuda support and has its own binaries i suppose.

I tried looking at the https://github.com/bottlesdevs/components/tree/main repo, and keeping this up seems simple enough, but it still needs some under-the-hood changes to bottles itself to actually show up as a option in the components section.

If anyone is interested in using this for that purpose i am all for it.... i just dont really have the time to figure out all the ins and outs of modifying the python scripts and whatnot that does the work in the bottles gui.

mheler commented 2 months ago

@SveSop can you please provide the exact steps to install this into a Bottles prefix? Is the bottles prefix where the game or the runner is?

Thanks

SveSop commented 1 month ago

It is in the PREFIX. Everything wine uses is in a "prefix". So if i create a "bottle" named "Test", for me it will be: $HOME/.var/app/com.usebottles.bottles/data/bottles/bottles/Test

Copy the file nvcuda-0.1/x32/nvcuda.dll -> $HOME/.var/app/com.usebottles.bottles/data/bottles/bottles/Test/drive_c/windows/syswow64 and nvcuda-0.1/x64/nvcuda.dll -> $HOME/.var/app/com.usebottles.bottles/data/bottles/bottles/Test/drive_c/windows/system32

PS. Change "Test" to the name of YOUR "bottle" ofc.

TiZ-HugLife commented 2 weeks ago

I've noticed that you can do an installation similar to the simplified one even with the regular nvidia-libs releases, but you have do a little copying/renaming. You don't seem to need any DLL overrides this way; I can use GPU Iray in Daz Studio just fine when I install it this way and remove the overrides that existed before.

Create two directories inside the directory of the nvidia-libs release: system32 and syswow64.
- Unintuitively, system32 is where the 64-bit libs go and syswow64 is where the 32-bit libs go.
Copy all three .dll.so files from lib64/wine/x86_64-unix and to system32, then rename them so they are just .dll files.
Copy nvapi64.dll and nvofapi64.dll from lib64/wine/x86_64-windows to system32 as well.
Copy nvcuda.dll.so from lib/wine/i386-unix to syswow64 and rename it so it is just a .dll file.
Copy nvapi.dll from lib/wine/i386-windows to syswow64 as well.
Copy system32 and syswow64 to your bottle's drive_c/windows directory.

It might be nice to create this directory structure in future releases for people to more easily install into their bottles, and to potentially integrate more easily into Bottles.

This does not install NVML. I know that the readme in nvidia-libs mentions that an extra CUDA device appears in Daz Studio without it, but that doesn't happen for me.

SveSop commented 2 weeks ago

The reason for using an override is when wine is compiled with said libraries. Some wine versions (especially older ones) have their own built-in version of nvapi/nvcuda, and if you just copy the dll's into the prefix without overrides it may end up loading the wine versions instead.

Proton-GE does use a stub-library for nvcuda in the steam version, but i am unsure if wine-GE does, and i have not really tested this overly much.

Less of a problem these days, but using an override by default does not hurt in any way even if it is not needed in most cases, so i think i would go for a recommendation of having it, in the case of a script implementation of this.

evertvorster commented 6 days ago

Hi there!

There is an update to this issue. To recap, I have a Insta360 X4 camera. The software that comes with it is Windows only. It leans heavily on the GPU to do encoding and decoding. What I first thought was missing CUDA support was something else. When installing Insta360 Studio into a vanilla bottle, and going into the settings app the Hardware Encode and Decode options were not detected and so not selected. Selecting these options does not make any change, playback of videos are choppy, and encoding takes a long time.

In troubleshooting this, it appeared that CUDA was enabled, but what was missing was nvcuvid/nvencodeapi

There has been an update in SveSop's repo, and now that installs the missing bits. https://github.com/SveSop/nvidia-libs/issues/25

When installing this into a Bottle, I now get full hardware encode and decode on my nvidia GPU in that bottle.

The GPU acceleration now works properly, video playback is snappy and encodes are lightning fast. It is my opinion that this package also would help other software that would like to use nvidia GPU to do hardware encoding and decoding. Stuff like Handbrake, or maybe even Davinci Resolve.

What would be super awesome is if this package was select-able in the Settings dialog of a bottle, rather than manually downloading and installing it.

What would it take to make this a reality?

bottlesdevs / Bottles