[Bug]: python3: undefined symbol: cudaRuntimeGetVersion

[Cinderella-Man]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

I updated my extensions today and dreambooth extension is causing the whole A1111 to fail on startup

Steps to reproduce the problem

1. Starting ./webui
2. Crashed upon startup

Commit and libraries

b4053defa6ae018b2ea56ac243aa55063f76fe0e

Command Line Arguments

No

Console logs

https://pastebin.com/27cCRsKY

Additional information

No response

[Matthieubmt]

same

[GNU-Linuxer]

same

[mateuspestana]

Same here.

[sbersier]

In my case, I found that the bitsandbytes module VERSION was the problem.

NOTE: I tested the following on linux Ubuntu 22.04 with CUDA 12.2 (but as far I understand it, the specific cuda version is not the point).

Before proceeding, check that bitsandbytes module is indeed the problem:

source /path/to/your/stable-diffusion-webui/venv/bin/activate

python -c "import bitsandbytes"

If it doesn't throw an error then forget about this post. Your problem is different. (NOTE: A warning is NOT an error,)

If it DOES throw an error then:

Here is how I solved it:

A) pip uninstall bitsandbytes

delete any bitsandbytes related folders:

rm -rf stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes*

B) Edit the stable-diffusion-webui/extensions/sd_dreambooth_extension/requirements.txt file:

I replaced (line 2):

bitsandbytes==0.35.4

with:

bitsandbytes==0.41.1

and saved it-

C) I needed to specify the CUDA version to use:

c.1) To get the cuda version:

nvcc --version

In my case, it returns:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jul_11_02:20:44_PDT_2023
Cuda compilation tools, release 12.2, V12.2.128
Build cuda_12.2.r12.2/compiler.33053471_0

c.2) So, I added the following line to my .bashrcfile in my $HOME folder (note the absence of decimal point in the version):

export BNB_CUDA_VERSION=122 (put here YOUR cuda version)

Save your .bashrc file

Then, in the terminal:

source ~/.bashrc

Test with:

echo $BNB_CUDA_VERSION

It should return your cuda version. In my case it returns; 122

D) Finally:

./webui.sh --xformers

It launches correctly and the Dreambooth tab is there.

Note that I haven't tried yet to train a Lora with this new configuration but everything seems OK...

[Mte90]

I changed also the postinstall.py that check the package version.

[sbersier]

@Mte90 Indeed a good idea. And it could also be a good idea to set the environment variable DREAMBOOTH_SKIP_INSTALL=True (in the bashrc file) in order not to put a mess in the install again.

[github-actions[bot]]

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Mte90]

well the issue is still there so closing doesn't make so much sense...