Google Colab upgraded CUDA in their base container images to 12.2, and now notebooks depending on bitsandbytes are all broken

[apurtell]

System Info

CUDA SETUP: Detected CUDA version 122 CUDA SETUP: Required library version not found: libbitsandbytes_cuda122_nocublaslt.so. Maybe you need to compile it from source?

Then bitsandbytes attempt to compile it automatically but setup.py gets confused when detecting paths and fails. And then the notebook fails.

CUDA SETUP: Something unexpected happened. Please compile from source: git clone git@github.com:TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=122_nomatmul python setup.py install CUDA SETUP: Setup Failed! CUDA SETUP: Something unexpected happened. Please compile from source: git clone git@github.com:TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=122_nomatmul python setup.py install

/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')} warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8013'), PosixPath('http'), PosixPath('//172.28.0.1')} warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-v100-s-uz348on3qd9l --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')} warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/datalab/web/pyright/typeshed-fallback/stdlib,/usr/local/lib/python3.10/dist-packages')} warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')} warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')} warn(msg) /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU! warn(msg)

Reproduction

Use anything on Colab that depends on bitsandbytes.

Expected behavior

After pip installs bitsandbytes from requirements, CUDA version "122" is detected and libbitsandbytes_cuda122_nocublaslt.so is available, is loaded successfully, and the notebook operates normally.

[younesbelkada]

Hi @apurtell Thanks for the issue! I quickly tried to install bnb on a free-tier google colab instance: https://colab.research.google.com/drive/1PO6Vy0ECeA-AL2wYa9iQV8YlcdS8pueb?usp=sharing are you using Gcolab pro?

[josemerinom]

!apt-get update !apt-get install cuda-toolkit-11-8 import os os.environ["LD_LIBRARY_PATH"] += ":" + "/usr/local/cuda-11/lib64" os.environ["LD_LIBRARY_PATH"] += ":" + "/usr/local/cuda-11.8/lib64"

[apurtell]

The above is not a solution. It could be a workaround, for some.

Native libraries compiled for various CUDA versions are shipped and selected by bitsandbytes according to detected version. What is missing here are native libraries compiled for recent CUDA versions. That is something that should be provided in new releases.

Alternatively if the existing shipped native libraries will still link and function as expected with recent CUDA versions, then the native library selection logic can be changed to attempt a fallback to whatever is most recent.

Falling back to compiling from source is definitely a reasonable last resort, but of course it’s not going to work wherever someone has (rightly) removed the compiler toolchain from container or VM images, which should be expected, because it is a very basic security hardening step.

[josemerinom]

1. I never said it was a solution
2. For me, I work at Google Colab, it is a solution

I hope you find the "solution"

[apurtell]

Come on, be professional. Is Reddit or X down?

[josemerinom]

I am a basic user

If I were a professional, I would read the comment, check what was proposed, and appreciate the comment, even if it is a partial solution, because if you publish a problem, it is because you do not have a solution, and there are people who want to help you

I had your problem and I solved it by reading a comment from another basic user on another github, and I implemented it and it worked where I work (google colab)

pd: I don't use fb/ig/tw, but I understand your sarcasm,

[jackiter]

!apt-get update !apt-get install cuda-toolkit-11-8 import os os.environ["LD_LIBRARY_PATH"] += ":" + "/usr/local/cuda-11/lib64" os.environ["LD_LIBRARY_PATH"] += ":" + "/usr/local/cuda-11.8/lib64"

Hi, can you tell me where in collab I have to paste this code, and instead of what? I worked with the ShivamShrirao colab. It was great but now I only get errors. thanks!