search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
140.96k stars 26.67k forks source link

[Bug]: python3: undefined symbol: cudaRuntimeGetVersion #12770

Open huuck opened 1 year ago

huuck commented 1 year ago

Is there an existing issue for this?

What happened?

Getting this when launching webui.sh --xformers:

Launching launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
Version: v1.5.2
Commit hash: c9c8485bc1e8720aba70f029d25cba1c4abf2b5c
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: cf086c536b141fc522ff11f6cffc8b7b12da04b9
Successfully installed accelerate-0.21.0 fastapi-0.94.1 gitpython-3.1.32 transformers-4.30.2

Does your project take forever to startup?
Repetitive dependency installation may be the reason.
Automatic1111's base project sets strict requirements on outdated dependencies.
If an extension is using a newer version, the dependency is uninstalled and reinstalled twice every startup.

[+] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.19.3 installed.
[+] transformers version 4.30.2 installed.
[+] bitsandbytes version 0.35.4 installed.

Launching Web UI with arguments: 
Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 85, in <module>
    from accelerate import __version__ as accelerate_version
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 131, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in <module>
    import bitsandbytes as bnb
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 113, in <module>
    lib = CUDASetup.get_instance().lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 109, in get_instance
    cls._instance.initialize()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 59, in initialize
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 125, in evaluate_cuda_setup
    cuda_version_string = get_cuda_version(cuda, cudart_path)
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 45, in get_cuda_version
    check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: cudaRuntimeGetVersion

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 39, in <module>
    main()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 35, in main
    start()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/modules/launch_utils.py", line 390, in start
    import webui
  File "/home/haxoman/dreambooth/stable-diffusion-webui/webui.py", line 39, in <module>
    import pytorch_lightning   # noqa: F401 # pytorch_lightning should be imported after torch, but it re-enables warnings on import so import once to disable them
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 35, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in <module>
    from torchmetrics import Metric
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 120, in <module>
    from torchmetrics.functional.text._deprecated import _bleu_score as bleu_score
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/__init__.py", line 50, in <module>
    from torchmetrics.functional.text.bert import bert_score  # noqa: F401
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py", line 23, in <module>
    from torchmetrics.functional.text.helper_embedding_metric import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/helper_embedding_metric.py", line 27, in <module>
    from transformers import AutoModelForMaskedLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizerBase
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1076, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):

Similar to this: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/12590 which was CLOSED for some arcane reason.

Steps to reproduce the problem

  1. Launch webui.sh with xformers
  2. App throws the error above

What should have happened?

App should run.

Version or Commit where the problem happens

1.5.2

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Linux

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

xformers

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--xformers

List of extensions

Dreambooth

Console logs

################################################################
Launching launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]
Version: v1.5.2
Commit hash: c9c8485bc1e8720aba70f029d25cba1c4abf2b5c
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: cf086c536b141fc522ff11f6cffc8b7b12da04b9
Successfully installed accelerate-0.21.0 fastapi-0.94.1 gitpython-3.1.32 transformers-4.30.2

Does your project take forever to startup?
Repetitive dependency installation may be the reason.
Automatic1111's base project sets strict requirements on outdated dependencies.
If an extension is using a newer version, the dependency is uninstalled and reinstalled twice every startup.

[+] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.19.3 installed.
[+] transformers version 4.30.2 installed.
[+] bitsandbytes version 0.35.4 installed.

Launching Web UI with arguments: 
Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 85, in <module>
    from accelerate import __version__ as accelerate_version
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 131, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in <module>
    import bitsandbytes as bnb
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 113, in <module>
    lib = CUDASetup.get_instance().lib
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 109, in get_instance
    cls._instance.initialize()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 59, in initialize
    binary_name, cudart_path, cuda, cc, cuda_version_string = evaluate_cuda_setup()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 125, in evaluate_cuda_setup
    cuda_version_string = get_cuda_version(cuda, cudart_path)
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 45, in get_cuda_version
    check_cuda_result(cuda, cudart.cudaRuntimeGetVersion(ctypes.byref(version)))
  File "/usr/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python3: undefined symbol: cudaRuntimeGetVersion

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 39, in <module>
    main()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/launch.py", line 35, in main
    start()
  File "/home/haxoman/dreambooth/stable-diffusion-webui/modules/launch_utils.py", line 390, in start
    import webui
  File "/home/haxoman/dreambooth/stable-diffusion-webui/webui.py", line 39, in <module>
    import pytorch_lightning   # noqa: F401 # pytorch_lightning should be imported after torch, but it re-enables warnings on import so import once to disable them
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 35, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 25, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 27, in <module>
    from torchmetrics import Metric
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 120, in <module>
    from torchmetrics.functional.text._deprecated import _bleu_score as bleu_score
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/__init__.py", line 50, in <module>
    from torchmetrics.functional.text.bert import bert_score  # noqa: F401
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/bert.py", line 23, in <module>
    from torchmetrics.functional.text.helper_embedding_metric import (
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchmetrics/functional/text/helper_embedding_metric.py", line 27, in <module>
    from transformers import AutoModelForMaskedLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizerBase
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1076, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/haxoman/dreambooth/stable-diffusion-webui/venv/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
python3: undefined symbol: cudaRuntimeGetVersion

Additional information

No response

huuck commented 1 year ago

On Ubuntu 22 with kernel 6.2.0, NVIDIA 535 drivers.

Fri Aug 25 10:57:22 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3080 Ti     Off | 00000000:0B:00.0  On |                  N/A |
|  0%   43C    P8              39W / 400W |    410MiB / 12288MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2155      G   /usr/lib/xorg/Xorg                          197MiB |
|    0   N/A  N/A      2469      G   /usr/bin/gnome-shell                         14MiB |
|    0   N/A  N/A      5493      G   ...irefox/2748/usr/lib/firefox/firefox      183MiB |
+---------------------------------------------------------------------------------------+
huuck commented 1 year ago

For the poor souls that had to bang their head against this. Adding export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcudart.so to webui-user.sh seemed to have fixed it. Probably the latest versions have some kind of library issues with CUDA.

Brennanzuz commented 1 year ago

For the poor souls that had to bang their head against this export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcudart.so to webui-user.sh seemed to have fixed it. Probably the latest versions have some kind of library issues with CUDA.

I have the exact same configuration and specifications as you, and appended the line export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcudart.so, but got this error setting the variable:

################################################################
Launching launch.py...
################################################################
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libcudart.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libcudart.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libcudart.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Version: v1.5.2
Commit hash: c9c8485bc1e8720aba70f029d25cba1c4abf2b5c
Installing requirements

image

On Ubuntu 22.04.1 as well, with Nvidia settings:

Fri Aug 25 22:42:19 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3080 ...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   46C    P0              N/A /  90W |     10MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1151      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+
huuck commented 1 year ago

what does locate libcudart.so give you?

Brennanzuz commented 1 year ago

what does locate libcudart.so give you?

/home/brennanzuz/.local/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/brennanzuz/.local/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12
/home/brennanzuz/.local/share/Trash/files/bark-venv/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/brennanzuz/Hunter_CC/yolov5/blue-venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/brennanzuz/Hunter_CC/yolov5/blue-venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12
/home/brennanzuz/Retrieval-based-Voice-Conversion-WebUI/rvc-webui/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/brennanzuz/TTS/tts-venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/envs/tortoise/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/envs/tortoise/lib/libcudart.so.11.7.99
/home/brennanzuz/anaconda3/envs/vcclient-dev/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/envs/vcclient-dev/lib/libcudart.so.11.8.89
/home/brennanzuz/anaconda3/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/lib/libcudart.so.11.8.89
/home/brennanzuz/anaconda3/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/pkgs/cuda-cudart-11.7.99-0/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/pkgs/cuda-cudart-11.7.99-0/lib/libcudart.so.11.7.99
/home/brennanzuz/anaconda3/pkgs/cuda-cudart-11.8.89-0/lib/libcudart.so.11.0
/home/brennanzuz/anaconda3/pkgs/cuda-cudart-11.8.89-0/lib/libcudart.so.11.8.89
/home/brennanzuz/dolly-v2-12b/venv/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0
/usr/lib/x86_64-linux-gnu/libcudart.so
huuck commented 1 year ago

try with any of the other files, it's probably a permissions issue. Do you run the WebUI inside a docker container or anything like that? Or as a low privileged user?

Brennanzuz commented 1 year ago

try with any of the other files, it's probably a permissions issue. Do you run the WebUI inside a docker container or anything like that? Or as a low privileged user?

Perfect! I changed it to export LD_PRELOAD=~/anaconda3/lib/libcudart.so.11.0 instead and it works when I run bash webui.sh. I still wonder what caused this cudaRuntimeGetVersion though. Still, thanks a bunch!

huuck commented 1 year ago

Alright so after closing it and firing up A1111 again it crashed with another random library error. What fixed it for good was this:

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcudart.so export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcudart.so.11.5.117 export LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH"

Seems to be some kind of dependency issue. Get your sorbet together, ML community :/

paboum commented 1 year ago

This seems to only occur with python3.10 whereas with python3.11 it's fine. Then however, dreambooth can't run with 3.11 because of https://github.com/pytorch/pytorch/issues/104918