Windows not supported

[mottosso]

To save fellow Windows users some time and download bandwidth, this project only runs on Linux due to (at least) one of the dependencies not supporting Windows.

• https://github.com/TimDettmers/bitsandbytes#requirements--installation

You'll receive an error upon running it.

python.exe .\demo.py --cfg-path .\eval_configs\minigpt4_eval.yaml
# Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?

The .so extension is for Linux only.

[cibernicola]

Trying with bitsandbytes version from https://github.com/oobabooga/text-generation-webui

I'm getting this:

(minigpt4) J:\gpt\MiniGPT-4>python demo.py --cfg-path eval_configs/minigpt4_eval.yaml
Initializing Chat
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: Loading binary J:\conda\envs\minigpt4\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.dll...
J:\conda\envs\minigpt4\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
Traceback (most recent call last):
  File "J:\gpt\MiniGPT-4\demo.py", line 57, in <module>
    model = model_cls.from_config(model_config).to('cuda:0')
  File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config
    model = cls(
  File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in __init__
    self.llama_model = LlamaForCausalLM.from_pretrained(
  File "J:\conda\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2740, in from_pretrained
    raise ValueError(
ValueError:
                        Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
                        the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
                        these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
                        `device_map` to `from_pretrained`. Check
                        https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu

[mottosso]

One step closer! However..

The installed version of bitsandbytes was compiled without GPU support

I would be weary of this; I expect a CPU version to take minutes to hours to days, compared to seconds.

[cibernicola]

Still searching and remixing :S Don't know why its saying that having in mind that https://github.com/oobabooga/text-generation-webui is working in CUDA

¬¬'

[cibernicola]

I guess is this: https://github.com/Vision-CAIR/MiniGPT-4/issues/4

[hufuzhipeng]

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so... argument of type 'WindowsPath' is not iterable C:\Users\zphu.conda\envs\minigpt4\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " This model has some weights that should be kept in higher precision, you need to upgrade accelerate to properly deal with them (pip install --upgrade accelerate). Traceback (most recent call last): File "E:\code\ai\MiniGPT-4\demo.py", line 57, in model = model_cls.from_config(model_config).to('cuda:0') File "E:\code\ai\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config model = cls( File "E:\code\ai\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in init self.llama_model = LlamaForCausalLM.from_pretrained( File "C:\Users\zphu.conda\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2740, in from_pretrained raise ValueError( ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom device_map to from_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.

(minigpt4) E:\code\ai\MiniGPT-4> (minigpt4) E:\code\ai\MiniGPT-4> (minigpt4) E:\code\ai\MiniGPT-4>python -m torch.utils.collect_env Collecting environment information... PyTorch version: 2.0.0+cpu Is debug build: False CUDA used to build PyTorch: Could not collect ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 企业版 GCC version: Could not collect Clang version: Could not collect CMake version: version 3.24.0-rc4 Libc version: N/A

Python version: 3.9.16 (main, Mar 8 2023, 10:39:24) [MSC v.1916 64 bit (AMD64)] (64-bit runtime) Python platform: Windows-10-10.0.19044-SP0 Is CUDA available: False CUDA runtime version: 11.7.64 CUDA_MODULE_LOADING set to: N/A GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090 Nvidia driver version: 516.01 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture=9 CurrentClockSpeed=4008 DeviceID=CPU0 Family=207 L2CacheSize=2048 L2CacheSpeed= Manufacturer=GenuineIntel MaxClockSpeed=4008 Name=Intel(R) Core(TM) i9-9900KS CPU @ 4.00GHz ProcessorType=3 Revision=

Versions of relevant libraries: [pip3] numpy==1.21.0 [pip3] torch==2.0.0 [pip3] torchaudio==0.12.1 [pip3] torchvision==0.15.1 [conda] blas 1.0 mkl [conda] cudatoolkit 11.3.1 h59b6b97_2 [conda] mkl 2021.4.0 haa95532_640 [conda] mkl-service 2.4.0 py39h2bbff1b_0 [conda] mkl_fft 1.3.1 py39h277e83a_0 [conda] mkl_random 1.2.2 py39hf11a4ad_0 [conda] numpy 1.23.5 py39h3b20f71_0 [conda] numpy-base 1.23.5 py39h4da318b_0 [conda] pytorch-mutex 1.0 cuda pytorch [conda] torch 2.0.0 pypi_0 pypi [conda] torchaudio 0.12.1 py39_cu113 pytorch [conda] torchvision 0.15.1 pypi_0 pypi

[thibaudart]

Still searching and remixing :S Don't know why its saying that having in mind that https://github.com/oobabooga/text-generation-webui is working in CUDA

¬¬'

Yes with that, we can have GPU support but still some error:

Any idea how to solve this:

Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom device_map to from_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.

I have 48gb of vram the GPU RAM must be enough!

[BANGBUP]

use windows precompiled version dlls from below. https://github.com/DeXtmL/bitsandbytes-win-prebuilt

and, you have to change some source code....

C:\ProgramData\Anaconda3\envs\minigpt4\Lib\site-packages\bitsandbytes\cuda_setup\main.py (my case...) about 370 line: if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None to if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None

and, about 126 line: self.lib = ct.cdll.LoadLibrary(binary_path) to self.lib = ct.cdll.LoadLibrary(str(binary_path))

then you can see no errors.

but my case, I still get an OOM problem... although i have 3090ti 24g...

(minigpt4) PS D:\pythonworkspace\MiniGPT-4> python demo.py --cfg-path eval_configs/minigpt4_eval.yaml Initializing Chat Loading VIT Loading VIT Done Loading Q-Former Loading Q-Former Done Loading LLAMA

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

binary_path = C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll This model has some weights that should be kept in higher precision, you need to upgrade accelerate to properly deal with them (pip install --upgrade accelerate). Loading checkpoint shards: 33%|███████████████████ | 1/3 [00:09<00:19, 9.92s/it] Traceback (most recent call last): File "D:\pythonworkspace\MiniGPT-4\demo.py", line 57, in model = model_cls.from_config(model_config).to('cuda:0') File "D:\pythonworkspace\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config model = cls( File "D:\pythonworkspace\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in init self.llama_model = LlamaForCausalLM.from_pretrained( File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2795, in from_pretrained ) = cls._load_pretrained_model( File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 3123, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 706, in _load_state_dict_into_meta_model set_module_8bit_tensor_to_device( File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\utils\bitsandbytes.py", line 78, in set_module_8bit_tensor_to_device new_value = bnb.nn.Int8Params(new_value, requires_grad=False, has_fp16_weights=has_fp16_weights).to(device) File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\bitsandbytes\nn\modules.py", line 196, in to return self.cuda(device) File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\bitsandbytes\nn\modules.py", line 159, in cuda B = self.data.contiguous().half().cuda(device) RuntimeError: CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 23.99 GiB total capacity; 7.83 GiB already allocated; 14.09 GiB free; 8.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF (minigpt4) PS D:\pythonworkspace\MiniGPT-4>

[cibernicola]

with the last version I'm reaching this point (without touching code):

`(miniGPT4) J:\gpt\MiniGPT-4>python demo.py --cfg-path eval_configs/minigpt4_eval.yaml Initializing Chat Loading VIT Loading VIT Done Loading Q-Former Loading Q-Former Done Loading LLAMA

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('J:/conda/envs/miniGPT4/bin')} warn(msg) J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: J:\conda\envs\miniGPT4 did not contain cudart64_110.dll as expected! Searching further paths... warn(msg) J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C:/Program Files (x86)/Java/jre1.8.0_321/bin'), WindowsPath('C:/Python38'), WindowsPath('L:/COLMAP-3.7-windows-cuda'), WindowsPath('C:/xampp/php'), WindowsPath('C:/Python38/Scripts'), WindowsPath('J:/conda/envs/miniGPT4/Library/mingw-w64/bin'), WindowsPath('J:/conda/envs/miniGPT4/bin'), WindowsPath('J:/conda/envs/miniGPT4/Library/usr/bin'), WindowsPath('J:/conda/envs/minigpt4/Lib/site-packages/cv2/../../x64/vc14/bin'), WindowsPath('C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64')} warn(msg) CUDA SETUP: CUDA runtime path found: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudart64_110.dll CUDA SETUP: Highest compute capability among GPUs detected: 8.6 CUDA SETUP: Detected CUDA version 117 CUDA SETUP: Loading binary J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll... Traceback (most recent call last): File "J:\gpt\MiniGPT-4\demo.py", line 57, in model = model_cls.from_config(model_config).to('cuda:0') File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config model = cls( File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in init self.llama_model = LlamaForCausalLM.from_pretrained( File "J:\conda\envs\miniGPT4\lib\site-packages\transformers\modeling_utils.py", line 2740, in from_pretrained raise ValueError( ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom device_map to from_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.

(miniGPT4) J:\gpt\MiniGPT-4>`

[Korner83]

It is working me on Win 10 GPU, just use this version for Bitsandbytes: https://github.com/Keith-Hon/bitsandbytes-windows.git

[cibernicola]

can you please describe a little bit about what you did?

[Korner83]

First I uninstalled the none-windows package of bitsandbytes and after that I installed this version in Anaconda: https://github.com/Keith-Hon/bitsandbytes-windows.git

You just need to have installed Git on your Windows

[zxcvbn114514]

First I uninstalled the none-windows package of bitsandbytes and after that I installed this version in Anaconda: https://github.com/Keith-Hon/bitsandbytes-windows.git

You just need to have installed Git on your Windows

I did it and it just showed Unable to create process using 'C:\Users\Ge Yunxiang.conda\envs\minigpt4\python.exe "C:\Users\Ge Yunxiang.conda\envs\minigpt4\Scripts\pip-script.py" install bitsandbytes'

[Korner83]

Check out this post: https://bobbyhadz.com/blog/python-fatal-error-in-launcher-unable-to-create-process-using-pip-exe

[zxcvbn114514]

Check out this post: https://bobbyhadz.com/blog/python-fatal-error-in-launcher-unable-to-create-process-using-pip-exe

I see.But I still dont know how to install bitsandbytes-windows with 'pip install bitsandbytes'.Shouldn't it become installing bitsandbytes from the source on the Internet rather than from the keith-hon's source?

[Korner83]

The official version is only for Linux. Only keith-hon's version of bitsandbyte supports Windows as far as I know. I had the same issue and was not working, because as a default it's installing wrong package (Linux version onto Windows) by running the command: pip install bitsandbyte

[zxcvbn114514]

The official version is only for Linux. Only keith-hon's version of bitsandbyte supports Windows as far as I know. I had the same issue and was not working, because as a default it's installing wrong package (Linux version onto Windows) by running the command: pip install bitsandbyte

OH,I understand .Thousand thanks man.But now I've encounted the same problem after successfully installed bitsandbytes-0.37.5-py3.9.egg.The picture below shows the problem.

[oldtreee]

正式版仅适用于 Linux。据我所知，只有 keith-hon 的 bitsandbyte 版本支持 Windows。我有同样的问题并且没有工作，因为默认情况下它通过运行命令安装错误的包（Linux 版本到 Windows）：pip install bitsandbyte

哦，我明白了。非常感谢。但是现在我在成功安装 bitsandbytes-0.37.5-py3.9.egg 后遇到了同样的问题。下图显示了问题。

我也遇到了同样的问题，你的解决了吗？

[zxcvbn114514]

正式版仅适用于 Linux。据我所知，只有 keith-hon 的 bitsandbyte 版本支持 Windows。我有同样的问题并且没有工作，因为默认情况下它通过运行命令安装错误的包（Linux 版本到 Windows）：pip install bitsandbyte

哦，我明白了。非常感谢。但是现在我在成功安装 bitsandbytes-0.37.5-py3.9.egg 后遇到了同样的问题。下图显示了问题。

我也遇到了同样的问题，你的解决了吗？

看看是不是虚拟环境的torch版本装成cpu了。 还有能加一下qq讨论一下吗，我是571130091

[latorc]

I got it working on Windows 10 (CUDA 118, RTX 4090). Install this version compatible with windows: conda install git pip pip uninstall bitsandbytes pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

binary_path: C:\ProgramData\miniconda3\envs\minigpt4\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll CUDA SETUP: Loading binary C:\ProgramData\miniconda3\envs\minigpt4\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll... Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:29<00:00, 9.74s/it] Loading LLAMA Done Load 4 training prompts Prompt Example

Human: Describe this image in detail. ###Assistant:

Load BLIP2-LLM Checkpoint: ./model/pretrained_minigpt4.pth Initialization Finished Running on local URL: http://127.0.0.1:7860

First I uninstalled the none-windows package of bitsandbytes and after that I installed this version in Anaconda: https://github.com/Keith-Hon/bitsandbytes-windows.git

You just need to have installed Git on your Windows