search

Bismuth-Consultancy-BV / MLOPs

Machine Learning Toolset for Houdini
BSD 3-Clause "New" or "Revised" License
371 stars 52 forks source link

Problem with the bitsandbytes. Models are not downloading. #118

Closed ASunren closed 1 year ago

ASunren commented 1 year ago

===================================BUG REPORT=================================== C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\cuda_setup\main.py:166: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/PROGRA~1/SIDEEF~1/Houdini 19.5.605/toolkit/bin'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/AlexDocuments/houdini19.5/MLOPs/MLOPs/data/models/diffusers/runwayml-_-stable-diffusion-v1-5'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/AlexDocuments/houdini19.5/modeler;C'), WindowsPath('/AlexDocuments/houdini19.5/MLOPs/MLOPs;&;C'), WindowsPath('/AlexDocuments/houdini19.5/MSLiveLink;C'), WindowsPath('C'), WindowsPath('/Program Files/Side Effects Software/sidefx_packages/SideFXLabs19.5;C'), WindowsPath('/AlexDocuments/houdini19.5/python_panels/jdb_houdinitoolkit;&'), WindowsPath('/PROGRA~1/SIDEEF~1/Houdini 19.5.605/packages/kinefx;C'), WindowsPath('/AlexDocuments/houdini19.5/MLOPs/MLOPs;C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/Program Files/Pixologic/ZBrush 2021/ZBrush.exe'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/z_cache'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/AlexDocuments/houdini19.5/qLib-master/otls/future;C'), WindowsPath('/AlexDocuments/houdini19.5/qLib-master/otls/base;C'), WindowsPath('/AlexDocuments/houdini19.5/qLib-master/otls/experimental;C'), WindowsPath('/AlexDocuments/houdini19.5/MOPS-master/otls;@/otls'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/AlexDocuments/houdini19.5/qLib-master/gallery;@/gallery'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/AlexDocuments/houdini19.5/MOPS-master/toolbar;@/toolbar'), WindowsPath('/AlexDocuments/houdini19.5/qLib-master/toolbar;C'), WindowsPath('/AlexDocuments/houdini19.5/toolbar;C'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/AlexDocuments/houdini19.5/qLib-master/scripts;@/scripts'), WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('//server/TEXTURE_LIBRARY/')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('C'), WindowsPath('/AlexDocuments/houdini19.5/poselib')} The following directories listed in your path were found to be non-existent: {WindowsPath('C')} The following directories listed in your path were found to be non-existent: {WindowsPath('/Users/Александр/untitled.hip'), WindowsPath('C')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 7.5. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local Traceback (most recent call last): File "C:\AlexDocuments/houdini19.5/MLOPs/MLOPs/scripts/python\mlops_utils.py", line 318, in on_accept ensure_huggingface_model_local(model_name, download_dir) File "C:\AlexDocuments/houdini19.5/MLOPs/MLOPs/scripts/python\mlops_utils.py", line 209, in ensure_huggingface_model_local from diffusers import StableDiffusionPipeline File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport__feature.py", line 142, in _import return original_import(name, *args, kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\diffusers__init.py", line 3, in from .configuration_utils import ConfigMixin File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport__feature__.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\diffusers\configuration_utils.py", line 34, in from .utils import ( File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature__.py", line 142, in _import return original_import(name, *args, kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\diffusers\utils__init.py", line 21, in from .accelerate_utils import apply_forward_hook File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport__feature__.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\diffusers\utils\accelerate_utils.py", line 24, in import accelerate File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\accelerate\init.py", line 3, in from .accelerator import Accelerator File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\accelerate\accelerator.py", line 35, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\accelerate\checkpointing.py", line 24, in from .utils import ( File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\accelerate\utils\init.py", line 131, in from .bnb import has_4bit_bnb_layers, load_and_quantize_model File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\accelerate\utils\bnb.py", line 42, in import bitsandbytes as bnb File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\init__.py", line 6, in from . import cuda_setup, utils, research File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\research\init.py", line 1, in from . import nn File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\research\nn\init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\research\nn\modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\optim\init.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "C:\Program Files\Side Effects Software\Houdini 19.5.605\python39\lib\site-packages-forced\shiboken2\files.dir\shibokensupport\feature__.py", line 142, in _import return original_import(name, *args, **kwargs) File "C:\AlexDocuments\houdini19.5\MLOPs\MLOPs\data\dependencies\python\bitsandbytes\cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
Ambrosiussen commented 1 year ago

Hi, please update MLOPs and reinstall dependencies. I believe this should be fixed now!

ajz3d commented 1 year ago

I believe I have the same problem, but on Debian 12.1. After installing requirements and restarting Houdini, any attempt to download a SD model throws the following stack trace. This comes from MLOPs version that I cloned two days ago.

===================================BUG REPORT===================================
/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/cuda_setup/main.py:166: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

  warn(msg)
================================================================================
The following directories listed in your path were found to be non-existent: {PosixPath('local/my_hostname'), PosixPath('@/tmp/.ICE-unix/2683,unix/my_hostname')}
The following directories listed in your path were found to be non-existent: {PosixPath('/org/freedesktop/DisplayManager/Session0')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/houdini19.5')}
The following directories listed in your path were found to be non-existent: {PosixPath('/org/freedesktop/DisplayManager/Seat0')}
The following directories listed in your path were found to be non-existent: {PosixPath('--install_base "/home/my_username/perl5"')}
The following directories listed in your path were found to be non-existent: {PosixPath('INSTALL_BASE=/home/my_username/perl5')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/.config/houdini19.5/packages/MLOPs/data/models')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/.config/houdini19.5/packages/MLOPs/data/models/diffusers/runwayml-_-stable-diffusion-v1-5')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/.config/houdini19.5/packages/MLOPs/data/models/cache')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/.config/houdini19.5/nvil-houdini-applink'), PosixPath('/home/my_username/.config/houdini19.5/dynamite'), PosixPath('&')}
The following directories listed in your path were found to be non-existent: {PosixPath('&')}
The following directories listed in your path were found to be non-existent: {PosixPath('/opt/meshroom/AliceVision')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/.config/houdini19.5/poselib')}
The following directories listed in your path were found to be non-existent: {PosixPath('/home/my_username/untitled.hip')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
DEBUG: Possible options found for libcudart.so: set()
CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Loading binary /home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/libbitsandbytes_cuda118.so...
libcusparse.so.11: cannot open shared object file: No such file or directory
CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected.
CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable
CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null
CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a
CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.
CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh
CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.
CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local
Traceback (most recent call last):
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/scripts/python/mlops_utils.py", line 381, in on_accept
    ensure_huggingface_model_local(model_name, download_dir)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/scripts/python/mlops_utils.py", line 272, in ensure_huggingface_model_local
    from diffusers import StableDiffusionPipeline
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/diffusers/__init__.py", line 3, in <module>
    from .configuration_utils import ConfigMixin
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/diffusers/configuration_utils.py", line 34, in <module>
    from .utils import (
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/diffusers/utils/__init__.py", line 21, in <module>
    from .accelerate_utils import apply_forward_hook
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/diffusers/utils/accelerate_utils.py", line 24, in <module>
    import accelerate
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/accelerate/utils/__init__.py", line 131, in <module>
    from .bnb import has_4bit_bnb_layers, load_and_quantize_model
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/accelerate/utils/bnb.py", line 42, in <module>
    import bitsandbytes as bnb
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/opt/hfs19.5.640/python/lib/python3.9/site-packages-forced/shiboken2/files.dir/shibokensupport/__feature__.py", line 142, in _import
    return original_import(name, *args, **kwargs)
  File "/home/my_username/.config/houdini19.5/packages/MLOPs/data/dependencies/python/bitsandbytes/cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them