Closed Andie-Squirrel closed 1 year ago
Looks like Bits and Bytes issue. See
https://github.com/oobabooga/text-generation-webui/issues/1068#issuecomment-1504869445 https://github.com/oobabooga/text-generation-webui/issues/1068#issuecomment-1506107242
Basically you will need a different bitsandbytes file, or need to edit the existing file to be correct. Also see: Install bitsandbytes for 8bit support (skip this on Linux) Install bitsandbytes (Windows only)
Download these 2 dll files: https://github.com/DeXtmL/bitsandbytes-win-prebuilt/raw/main/libbitsandbytes_cpu.dll https://github.com/DeXtmL/bitsandbytes-win-prebuilt/raw/main/libbitsandbytes_cuda116.dll Move those files into C:\Users\xxx\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\ Now edit bitsandbytes\cuda_setup\main.py with these: Change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path)) two times in the file. Then replace if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None with if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None
from https://rentry.org/llama-tard-v2#bonus-3-4bit-llama-basic-setup
Hey MrVirus, thank you very much. I was wondering if bitsandbytes was the source of the problem. Appreciate your detailed solution, I will try it and report back if it works.
It appears I do not have a 'miniconda3' directory in my C:\Users[name]\ folder. Do I need to have miniconda installed?
Looks like Bits and Bytes issue. See
#1068 (comment) #1068 (comment)
Basically you will need a different bitsandbytes file, or need to edit the existing file to be correct. Also see: Install bitsandbytes for 8bit support (skip this on Linux) Install bitsandbytes (Windows only)
Download these 2 dll files: https://github.com/DeXtmL/bitsandbytes-win-prebuilt/raw/main/libbitsandbytes_cpu.dll https://github.com/DeXtmL/bitsandbytes-win-prebuilt/raw/main/libbitsandbytes_cuda116.dll Move those files into C:\Users\xxx\miniconda3\envs\textgen\lib\site-packages\bitsandbytes Now edit bitsandbytes\cuda_setup\main.py with these: Change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path)) two times in the file. Then replace if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None with if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None
from https://rentry.org/llama-tard-v2#bonus-3-4bit-llama-basic-setup
I have been searching but I don't believe I have the required directory. If this specifically is an issue with an unrelated piece of software, should this be closed as it doesn't relate to oobabooga?
Alright, it's been fixed for me after figuring out where the libbitsandbytes_cuda116.dll belongs. Still having an issue with CUDA but that's a different issue entirely.
Describe the bug
I have been running into issues with running the start-webuil.bat file. After running install.bat to upgrade, I get a TypeError and the installer halts. Prior to this, after selecting a model and inputting a prompt the program ran into another error with torch and bitsandbytes. After following the instructions in the comments to run
pip install -i https://test.pypi.org/simple/ bitsandbytes-cuda113
, I re-ran the webui and got this:and the log:
Starting the web UI... Traceback (most recent call last): File "A:\oogabooga\oobabooga-windows\oobabooga-windows\text-generation-webui\server.py", line 28, in <module> from modules import api, chat, shared, training, ui File "A:\oogabooga\oobabooga-windows\oobabooga-windows\text-generation-webui\modules\training.py", line 12, in <module> from peft import (LoraConfig, get_peft_model, get_peft_model_state_dict, File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\site-packages\peft\__init__.py", line 22, in <module> from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CONFIG_MAPPING, get_peft_config, get_peft_model File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\site-packages\peft\mapping.py", line 16, in <module> from .peft_model import ( File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\site-packages\peft\peft_model.py", line 31, in <module> from .tuners import AdaLoraModel, LoraModel, PrefixEncoder, PromptEmbedding, PromptEncoder File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\site-packages\peft\tuners\__init__.py", line 20, in <module> from .lora import LoraConfig, LoraModel File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\site-packages\peft\tuners\lora.py", line 39, in <module> import bitsandbytes as bnb File "C:\Users\[name]\AppData\Roaming\Python\Python310\site-packages\bitsandbytes\__init__.py", line 6, in <module> from .nn import modules File "C:\Users\[name]\AppData\Roaming\Python\Python310\site-packages\bitsandbytes\nn\__init__.py", line 5, in <module> from .modules import StableEmbedding, Linear8bit, Linear8bitLt, Int8Params File "C:\Users\[name]\AppData\Roaming\Python\Python310\site-packages\bitsandbytes\nn\modules.py", line 15, in <module> from bitsandbytes.optim import GlobalOptimManager File "C:\Users\[name]\AppData\Roaming\Python\Python310\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module> from bitsandbytes.cextension import COMPILED_WITH_CUDA File "C:\Users\[name]\AppData\Roaming\Python\Python310\site-packages\bitsandbytes\cextension.py", line 5, in <module> lib = ct.cdll.LoadLibrary(os.path.dirname(__file__) + '/libbitsandbytes.so') File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\ctypes\__init__.py", line 452, in LoadLibrary return self._dlltype(name) File "A:\oogabooga\oobabooga-windows\oobabooga-windows\installer_files\env\lib\ctypes\__init__.py", line 374, in __init__ self._handle = _dlopen(self._name, mode) OSError: [WinError 193] %1 is not a valid Win32 application Press any key to continue . . .
To fix this, I ran the installer which yielded the log I will include in the Logs section.
Additionally, another possible cause for this is that I have two installations of Python (3.9 and 3.10), but both are added to PATH.
This may be a duplicate of https://github.com/oobabooga/text-generation-webui/issues/1037#issue-1661748221
Is there an existing issue for this?
Reproduction
Running install.bat in the oobabooga-windows directory.
Screenshot
Logs
System Info