search

bitsandbytes-foundation / bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.
https://huggingface.co/docs/bitsandbytes/main/en/index
MIT License
6.27k stars 627 forks source link

CUDA Error #65

Closed tarushbansal closed 10 months ago

tarushbansal commented 2 years ago

CUDA SETUP: CUDA runtime path found: /usr/local/Cluster-Apps/cuda/8.0/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 6.0 CUDA SETUP: CUDA version lower than 11 are currenlty not supported for LLM.int8(). You will be only to use 8-bit optimizers and quantization routines!! CUDA SETUP: Detected CUDA version 80 CUDA SETUP: TODO: compile library for specific version: libbitsandbytes_cuda80_nocublaslt.so CUDA SETUP: Defaulting to libbitsandbytes.so... CUDA SETUP: CUDA detection failed. Either CUDA driver not installed, CUDA not installed, or you have multiple conflicting CUDA libraries! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. Traceback (most recent call last): File "fine_tune_gpt-j.py", line 15, in from bitsandbytes.functional import quantize_blockwise, dequantize_blockwise File "/home/tb662/env/myenv/lib64/python3.6/site-packages/bitsandbytes/init.py", line 6, in from .autograd._functions import ( File "/home/tb662/env/myenv/lib64/python3.6/site-packages/bitsandbytes/autograd/_functions.py", line 5, in import bitsandbytes.functional as F File "/home/tb662/env/myenv/lib64/python3.6/site-packages/bitsandbytes/functional.py", line 13, in from .cextension import COMPILED_WITH_CUDA, lib File "/home/tb662/env/myenv/lib64/python3.6/site-packages/bitsandbytes/cextension.py", line 41, in lib = CUDALibrary_Singleton.get_instance().lib File "/home/tb662/env/myenv/lib64/python3.6/site-packages/bitsandbytes/cextension.py", line 37, in get_instance cls._instance.initialize() File "/home/tb662/env/myenv/lib64/python3.6/site-packages/bitsandbytes/cextension.py", line 27, in initialize raise Exception('CUDA SETUP: Setup Failed!') Exception: CUDA SETUP: Setup Failed!

walter-grace commented 2 years ago

Getting a similar error: /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:106: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths... f'{candidate_env_vars["LD_LIBRARY_PATH"]} did not contain ' /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('["--ip=172.28.0.2"],"debugAdapterMultiplexerPath"'), PosixPath('{"kernelManagerProxyPort"'), PosixPath('true}'), PosixPath('"/usr/local/bin/dap_multiplexer","enableLsp"'), PosixPath('"172.28.0.3","jupyterArgs"'), PosixPath('6000,"kernelManagerProxyHost"')} "WARNING: The following directories listed in your path were found to " /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')} "WARNING: The following directories listed in your path were found to " /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')} "WARNING: The following directories listed in your path were found to " CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA SETUP: Detected CUDA version 112 CUDA SETUP: Loading binary /usr/local/lib/python3.7/dist-packages/bitsandbytes/libbitsandbytes_cuda112.so... Caching latents: 100% 50/50 [00:10<00:00, 4.82it/s] Steps: 0% 0/1000 [00:00<?, ?it/s]Traceback (most recent call last): File "train_dreambooth.py", line 646, in main() File "train_dreambooth.py", line 591, in main noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/usr/local/lib/python3.7/dist-packages/accelerate/utils/operations.py", line 507, in call return convert_to_fp32(self.model_forward(*args, *kwargs)) File "/usr/local/lib/python3.7/dist-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast return func(args, kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d_condition.py", line 327, in forward upsample_size=upsample_size, File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 1149, in forward hidden_states = attn(hidden_states, context=encoder_hidden_states) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 169, in forward hidden_states = block(hidden_states, context=context) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(input, kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 218, in forward hidden_states = self.attn1(self.norm1(hidden_states)) + hidden_states File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 293, in forward hidden_states = self._attention(query, key, value) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 302, in _attention attention_probs = attention_scores.softmax(dim=-1) RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 12.29 GiB already allocated; 945.75 MiB free; 12.57 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Steps: 0% 0/1000 [00:02<?, ?it/s] Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'train_dreambooth.py', '--pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4', '--instance_data_dir=/content/data/NicoInput', '--class_data_dir=/content/data/person', '--output_dir=/content/drive/MyDrive/stable_diffusion_weights/sks', '--with_prior_preservation', '--prior_loss_weight=1.0', '--instance_prompt=realnico', '--class_prompt=person', '--seed=1337', '--resolution=512', '--train_batch_size=1', '--mixed_precision=fp16', '--use_8bit_adam', '--gradient_accumulation_steps=1', '--learning_rate=5e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=50', '--sample_batch_size=4', '--max_train_steps=1000']' returned non-zero exit status

Titus-von-Koeller commented 2 years ago

Dear Walter, your error is not similar. The bitsandbytes debug information is printed out for ANY error. Your error is due to an exception in the external accelerate library. If you read what's written there, you'll see the reason for the error:

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 12.29 GiB already allocated; 945.75 MiB free; 12.57 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Steps: 0% 0/1000 [00:02<?, ?it/s]

On Wed, Oct 19, 2022, 01:17 Walter Grace @.***> wrote:

Getting a similar error: /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:106: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths... f'{candidate_env_vars["LD_LIBRARY_PATH"]} did not contain ' /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('["--ip=172.28.0.2"],"debugAdapterMultiplexerPath"'), PosixPath('{"kernelManagerProxyPort"'), PosixPath('true}'), PosixPath('"/usr/local/bin/dap_multiplexer","enableLsp"'), PosixPath('"172.28.0.3","jupyterArgs"'), PosixPath('6000,"kernelManagerProxyHost"')} "WARNING: The following directories listed in your path were found to " /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')} "WARNING: The following directories listed in your path were found to " /usr/local/lib/python3.7/dist-packages/bitsandbytes/cuda_setup/paths.py:28: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')} "WARNING: The following directories listed in your path were found to " CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.5 CUDA SETUP: Detected CUDA version 112 CUDA SETUP: Loading binary /usr/local/lib/python3.7/dist-packages/bitsandbytes/libbitsandbytes_cuda112.so... Caching latents: 100% 50/50 [00:10<00:00, 4.82it/s] Steps: 0% 0/1000 [00:00<?, ?it/s]Traceback (most recent call last): File "train_dreambooth.py", line 646, in main() File "train_dreambooth.py", line 591, in main noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/usr/local/lib/python3.7/dist-packages/accelerate/utils/operations.py", line 507, in call return convert_to_fp32(self.model_forward(*args, *kwargs)) File "/usr/local/lib/python3.7/dist-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast return func(args, kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_2d_condition.py", line 327, in forward upsample_size=upsample_size, File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/unet_blocks.py", line 1149, in forward hidden_states = attn(hidden_states, context=encoder_hidden_states) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, *kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 169, in forward hidden_states = block(hidden_states, context=context) File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(input, kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 218, in forward hidden_states = self.attn1(self.norm1(hidden_states)) + hidden_states File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 293, in forward hidden_states = self._attention(query, key, value) File "/usr/local/lib/python3.7/dist-packages/diffusers/models/attention.py", line 302, in _attention attention_probs = attention_scores.softmax(dim=-1) RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 14.76 GiB total capacity; 12.29 GiB already allocated; 945.75 MiB free; 12.57 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Steps: 0% 0/1000 [00:02<?, ?it/s] Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'train_dreambooth.py', '--pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4', '--instance_data_dir=/content/data/NicoInput', '--class_data_dir=/content/data/person', '--output_dir=/content/drive/MyDrive/stable_diffusion_weights/sks', '--with_prior_preservation', '--prior_loss_weight=1.0', '--instance_prompt=realnico', '--class_prompt=person', '--seed=1337', '--resolution=512', '--train_batch_size=1', '--mixed_precision=fp16', '--use_8bit_adam', '--gradient_accumulation_steps=1', '--learning_rate=5e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=50', '--sample_batch_size=4', '--max_train_steps=1000']' returned non-zero exit status

— Reply to this email directly, view it on GitHub https://github.com/TimDettmers/bitsandbytes/issues/65#issuecomment-1283124124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFBEOZ5ZX2SR6ASHXGOL63WD4VYFANCNFSM6AAAAAARINNU5A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

tarushbansal commented 2 years ago

Any suggestions on my issue? I am using my university HPC cluster so I can't really make changes to the CUDA libraries on the cluster nodes.

TimDettmers commented 2 years ago

@tarushbansal there are two issues:

  1. Your GPU has compute capability of 6.0 which is currently does not support int8 matrix multiplication (you need a newer GPU). 8-bit optimizer are still supported.
  2. You can use the cuda_install.sh script to install CUDA locally.

For installing CUDA locally, you can follow these instructions:

self.add_log_entry('CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA.')
self.add_log_entry('CUDA SETUP: Solution 2a): Download CUDA install script: wget https://github.com/TimDettmers/bitsandbytes/blob/main/cuda_install.sh')
self.add_log_entry('CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO.')
self.add_log_entry('CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local')

After following these instructions, make sure the CUDA install path is added the the $LD_LIBRARY_PATH variable in your .bashrc file.

swumagic commented 1 year ago

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang） 1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui 2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.