search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
39.9k stars 5.23k forks source link

Web UI not starting #936

Closed Dan00tz closed 11 months ago

Dan00tz commented 1 year ago

Describe the bug

image

Default installation without tampering the launch options.

Is there an existing issue for this?

Reproduction

Default installation.

Screenshot

No response

Logs

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: CUDA runtime path found: F:\GPT\oobabooga-windows\installer_files\env\bin\cudart64_110.dll
CUDA SETUP: Highest compute capability among GPUs detected: 6.1
CUDA SETUP: Detected CUDA version 117
F:\GPT\oobabooga-windows\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
  warn(msg)
CUDA SETUP: Loading binary F:\GPT\oobabooga-windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117_nocublaslt.dll...
Loading anon8231489123_vicuna-13b-GPTQ-4bit-128g...
Traceback (most recent call last):
  File "F:\GPT\oobabooga-windows\text-generation-webui\server.py", line 302, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "F:\GPT\oobabooga-windows\text-generation-webui\modules\models.py", line 102, in load_model
    model = load_quantized(model_name)
  File "F:\GPT\oobabooga-windows\text-generation-webui\modules\GPTQ_loader.py", line 135, in load_quantized
    model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, kernel_switch_threshold=threshold)
  File "F:\GPT\oobabooga-windows\text-generation-webui\modules\GPTQ_loader.py", line 54, in _load_quant
    make_quant(**make_quant_kwargs)
  File "F:\GPT\oobabooga-windows\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 446, in make_quant
    make_quant(child, names, bits, groupsize, faster, name + '.' + name1 if name != '' else name1, kernel_switch_threshold=kernel_switch_threshold)
  File "F:\GPT\oobabooga-windows\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 446, in make_quant
    make_quant(child, names, bits, groupsize, faster, name + '.' + name1 if name != '' else name1, kernel_switch_threshold=kernel_switch_threshold)
  File "F:\GPT\oobabooga-windows\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 446, in make_quant
    make_quant(child, names, bits, groupsize, faster, name + '.' + name1 if name != '' else name1, kernel_switch_threshold=kernel_switch_threshold)
  [Previous line repeated 1 more time]
  File "F:\GPT\oobabooga-windows\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 443, in make_quant
    module, attr, QuantLinear(bits, groupsize, tmp.in_features, tmp.out_features, faster=faster, kernel_switch_threshold=kernel_switch_threshold)
  File "F:\GPT\oobabooga-windows\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 154, in __init__
    'qweight', torch.zeros((infeatures // 32 * bits, outfeatures), dtype=torch.int)
RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 35389440 bytes.
Press any key to continue . . .

System Info

GPU GTX 1070
Windows 11
16GB RAM
I7-7700K
johnswan commented 1 year ago

I have the same issue with 2080Ti with 11Gb VRAM and 32Gb of RAM.

RAM and GPU Memory in Task Manager don't spike at all,

ehamawy commented 1 year ago

same issue, bump

CyrisXD commented 1 year ago

Same issue here, tried various fixes but nothing seems to work. Trying to load the gpt-x-alpaca-13b-native-4bit-128g-cuda.pt model.

32GB RAM RTX 3070 8GB

Samogub commented 1 year ago

same issue 1660 ti 6gb 16ram i think i cant launch cuz 6gb vram =)

skippyssk commented 1 year ago

I don't know how many people this will help, but I got the error because I was running Stable Diffusion at the same time.

jllllll commented 1 year ago

That model requires at least 9gb of free VRAM. If you have that, try adding --pre_layer 36.

The model has 40 layers and each layer is about 0.222 gb in size. --pre_layer determines the number of layers to put in VRAM.

CyrisXD commented 1 year ago

@jllllll Thanks, but --pre_layer 4 didn't work for me.

jllllll commented 1 year ago

@CyrisXD You will need a larger number due to your GPU not having enough VRAM: --pre_layer 9

CyrisXD commented 1 year ago

@jllllll Mmm doesn't work either. I've tried everything I can find and just can't get it to work.

This is how I'm trying to run it:

call python server.py --auto-devices --chat --pre_layer 9 --wbits 4 --groupsize 128 --model gpt4-x-alpaca-13b-native-4bit-128g

jllllll commented 1 year ago

@CyrisXD Oops I got it backwards. Also, remove --auto-devices.

--pre_layer 31

CyrisXD commented 1 year ago

@jllllll Unfortunately same result. I might just not be able to run this model with 8GB VRAM.

not enough memory: you tried to allocate 655380480 bytes.

jllllll commented 1 year ago

@CyrisXD The model has 40 layers and each layer is about 0.222 gb in size. --pre_layer determines the number of layers to put in VRAM.

Check task manager to see how much VRAM you have available. This could also just be a bug.

CyrisXD commented 1 year ago

@jllllll It may be a bug, good point. I notice the VRAM doesn't jump at all. The script just throws this error: 

Loading gpt4-x-alpaca-13b-native-4bit-128g...
Loading model ...
Traceback (most recent call last):
  File "D:\ChatGPT Models\text-generation-webui\server.py", line 302, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "D:\ChatGPT Models\text-generation-webui\modules\models.py", line 102, in load_model
    model = load_quantized(model_name)
  File "D:\ChatGPT Models\text-generation-webui\modules\GPTQ_loader.py", line 132, in load_quantized
    model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, shared.args.pre_layer)
  File "D:\ChatGPT Models\text-generation-webui\repositories\GPTQ-for-LLaMa\llama_inference_offload.py", line 228, in load_quant
    model.load_state_dict(torch.load(checkpoint))
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\serialization.py", line 809, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\serialization.py", line 1172, in _load
    result = unpickler.load()
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\serialization.py", line 1142, in persistent_load
    typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\serialization.py", line 1112, in load_tensor
    storage = zip_file.get_storage_from_record(name, numel, torch.UntypedStorage)._typed_storage()._untyped_storage
RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 655380480 bytes.
Press any key to continue . . .
jllllll commented 1 year ago

@CyrisXD Open micromama-cmd.bat and enter this command: python -m torch.utils.collect_env

Post the output. It will give details on your Torch installation.

CyrisXD commented 1 year ago

@jllllll Here is the output: 

Collecting environment information...
Traceback (most recent call last):
  File "D:\ChatGPT Models\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\ChatGPT Models\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\utils\collect_env.py", line 602, in <module>
    main()
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\utils\collect_env.py", line 585, in main
    output = get_pretty_env_info()
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\utils\collect_env.py", line 580, in get_pretty_env_info
    return pretty_str(get_env_info())
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\utils\collect_env.py", line 422, in get_env_info
    pip_version, pip_list_output = get_pip_packages(run_lambda)
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\utils\collect_env.py", line 394, in get_pip_packages
    out = run_with_pip(sys.executable + ' -mpip')
  File "D:\ChatGPT Models\installer_files\env\lib\site-packages\torch\utils\collect_env.py", line 382, in run_with_pip
    for line in out.splitlines()
AttributeError: 'NoneType' object has no attribute 'splitlines'
jllllll commented 1 year ago

@CyrisXD Well, that shouldn't have happened. I would suggest reinstalling in a new folder (without spaces) using the latest installer: https://github.com/oobabooga/one-click-installers

You could also try this command to reinstall Torch:

python -m pip install https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl --force-reinstall
shalevc1098 commented 1 year ago

i have this issue either

CyrisXD commented 1 year ago

@jllllll Ok after reinstalling Torch I am able to run micromama-cmd.bat and here is the new output:

(D:\ChatGPT Models\installer_files\env) D:\ChatGPT Models>python3 -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 2.0.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr  5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070
Nvidia driver version: 528.24
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=2100
DeviceID=CPU0
Family=198
L2CacheSize=10240
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=2100
Name=12th Gen Intel(R) Core(TM) i7-12700
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] torch==2.0.0+cu117
[conda] Could not collect

If nothing is obvious here, then I'll try a full re-install. Thank you for all the help so far.

jllllll commented 1 year ago

@CyrisXD If reinstalling Torch fixed that command, then your Torch install must have been broken somehow. I would try running the model again and seeing if it works now. If not, then reinstall.

CyrisXD commented 1 year ago

@jllllll Nope, didn't work. Tried reinstalling from scratch and same thing, not enough memory: you tried to allocate 13107200 bytes.

I'm out of ideas at this stage.

johnswan commented 1 year ago

Running python -m torch.utils.collect_env showed that I had [pip3] torch==2.0.0 instead of [pip3] torch==2.0.0+cu117 so I ran python -m pip install https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl --force-reinstall as suggested by @jllllll

But I still get RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 141557760 bytes.

Now I have 11Gb VRAM on a 2080ti, that's enough for this model, right?

Why is Pytorch messing around with a CPU allocator, rather than GPU?

Why is it unable to allocate enough memory, when there is 32Gb RAM and plenty is free?

And there is nothing spiking in Task Manager, so this must be a bug?

johnswan commented 1 year ago

I've reinstalled everything over and over again.

No change.

Who actually has this working?

jllllll commented 1 year ago

It works for me. I've tried all sorts of things to try and figure this out. All GPTQ models load fine on my system. I even tried updating GPTQ-for-LLaMa to the latest version. Still can't get it to produce this error.

If you want to try updating to the latest GPTQ-for-LLaMa, then open micromamba-cmd.bat and enter these in order:

cd ".\text-generation-webui\repositories"
rd /q /s ".\GPTQ-for-LLaMa\" >nul
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa.git -b cuda
cd GPTQ-for-LLaMa
git revert -n b55bd46b602ad497e0cbfe2ec2c7cce80013cebd
python -m pip install -r requirements.txt
python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/610fdae6588c2b17bcf2726cacaaf795cd45077e/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl --force-reinstall

If you want to try compiling GPTQ-for-LLaMa yourself, then replace the last command with this:

python -m pip uninstall quant_cuda
python setup_cuda.py install
jllllll commented 1 year ago

Also, watch your RAM in task manager as you try to load the model. It may be loading to RAM first before transferring to VRAM.

johnswan commented 1 year ago

@jllllll

Thanks for your continued help, it's appreciated.

Also, watch your RAM in task manager as you try to load the model. It may be loading to RAM first before transferring to VRAM.

As I mentioned before, nothing spikes in Task Manager, RAM remains flat as a pancake with 23Gb available, as does VRAM with 11Gb. I don't think it even attempts to allocate memory, it just fails.

What are your results for python -m torch.utils.collect_env?

johnswan commented 1 year ago

I loaded up the project in VS Code and now I can see a bunch of warnings that were not visible when running start-webui.bat.

This one looks important! \ow2\installer_files\env\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. "

Any idea? I found a CUDA version here but the install instructions look a bit hairy: https://pypi.org/project/bitsandbytes-cuda117/

johnswan commented 1 year ago

Installed the other bitsandbytes: pip install bitsandbytes-cuda117

But now I get:

[WinError 193] %1 is not a valid Win32 application
  File "C:\Users\John\Downloads\ow2\text-generation-webui\modules\training.py", line 12, in <module>
    from peft import (LoraConfig, get_peft_model, get_peft_model_state_dict,
  File "C:\Users\John\Downloads\ow2\text-generation-webui\server.py", line 18, in <module>
    from modules import api, chat, shared, training, ui
OSError: [WinError 193] %1 is not a valid Win32 application

What isn’t a valid Win32 application?

Look in C:\Users\John\Downloads\ow2\installer_files\env\lib\site-packages\bitsandbytes\functional.py:
lib = ct.cdll.LoadLibrary(os.path.dirname(__file__) + '/libbitsandbytes.so')

import os
os.path.dirname(__file__)
'C:\\Users\\John\\Downloads\\ow2/text-generation-webui\\modules'
__file__
'C:\\Users\\John\\Downloads\\ow2/text-generation-webui\\modules\\training.py'

No libbitsandbytes.so here, just Python files:

 Directory of C:\Users\John\Downloads\ow2\text-generation-webui\modules

09/04/2023  14:23    <DIR>          .
09/04/2023  14:23    <DIR>          ..
09/04/2023  13:59             1,074 api.py
09/04/2023  13:59             3,023 callbacks.py
09/04/2023  13:59            18,510 chat.py
09/04/2023  13:59             2,590 deepspeed_parameters.py
09/04/2023  13:59             2,447 extensions.py
09/04/2023  13:59             6,191 GPTQ_loader.py
09/04/2023  13:59             7,148 html_generator.py
09/04/2023  13:59             2,544 llamacpp_model.py
09/04/2023  13:59             1,866 llamacpp_model_alternative.py
09/04/2023  13:59             1,397 LoRA.py
09/04/2023  13:59            10,331 models.py
09/04/2023  13:59             2,480 RWKV.py
09/04/2023  13:59             8,627 shared.py
09/04/2023  13:59            12,137 text_generation.py
09/04/2023  13:59            17,217 training.py
09/04/2023  13:59             1,328 ui.py
09/04/2023  14:25    <DIR>          __pycache__
              16 File(s)         98,910 bytes
               3 Dir(s)  552,972,828,672 bytes free

Howeverlibbitsandbytes.so is here:

(C:\Users\John\Downloads\ow2\installer_files\env) C:\Users\John\Downloads\ow2\installer_files\env\lib\site-packages>dir ..\..\..\..\..\ow2\installer_files\env\lib\site-packages\bitsandbytes
 Volume in drive C has no label.
 Volume Serial Number is BAB1-B220

 Directory of C:\Users\John\Downloads\ow2\installer_files\env\lib\site-packages\bitsandbytes

09/04/2023  16:14    <DIR>          .
09/04/2023  16:14    <DIR>          ..
09/04/2023  13:59    <DIR>          autograd
09/04/2023  13:59             1,254 cextension.py
09/04/2023  13:59    <DIR>          cuda_setup
09/04/2023  16:14            32,648 functional.py
09/04/2023  16:14        10,371,720 libbitsandbytes.so
09/04/2023  13:59            91,648 libbitsandbytes_cpu.dll
09/04/2023  13:59            82,238 libbitsandbytes_cpu.exp
09/04/2023  13:59           133,088 libbitsandbytes_cpu.lib
09/04/2023  13:59         6,373,888 libbitsandbytes_cuda117.dll
09/04/2023  13:59           156,208 libbitsandbytes_cuda117.exp
09/04/2023  13:59           261,676 libbitsandbytes_cuda117.lib
09/04/2023  13:59        21,005,824 libbitsandbytes_cuda117_nocublaslt.dll
09/04/2023  13:59           154,563 libbitsandbytes_cuda117_nocublaslt.exp
09/04/2023  13:59           265,802 libbitsandbytes_cuda117_nocublaslt.lib
09/04/2023  16:14    <DIR>          nn
09/04/2023  16:14    <DIR>          optim
09/04/2023  13:59               709 utils.py
09/04/2023  16:14               642 __init__.py
09/04/2023  13:59             2,213 __main__.py
09/04/2023  16:14    <DIR>          __pycache__
              15 File(s)     38,934,121 bytes
               7 Dir(s)  552,414,912,512 bytes free

Would be easy enough to make a symlink on Linux, but this is Windows…

Any ideas?

johnswan commented 1 year ago

I've spent way too long on this, fixed some problems, tried to help others, but I'm about to give up now.

@jllllll You say you have it working, but maybe we are not talking about the same thing.

I'm trying to run gpt-x-alpaca-13b-native-4bit-128g with a 2080Ti/12Gb with i7/32Gb/Win10

Do you have gpt-x-alpaca-13b-native-4bit-128g running on a machine with similar specs?

Or with 12Gb, do I just not have enough VRAM to run this?

Thanks again for your help.

jllllll commented 1 year ago

@johnswan bitsandbytes does not support Windows. The installer uses a custom Windows-compatible version. If you were not using the latest installer, then you may not have gotten that version.

johnswan commented 1 year ago

@johnswan bitsandbytes does not support Windows. The installer uses a custom Windows-compatible version. If you were not using the latest installer, then you may not have gotten that version.

Which installer?

Could you send a link?

Because I'm sure I used whatever the "one-click-installer" link was in the README last night, and I don't think it's changed since.

jllllll commented 1 year ago

@johnswan https://github.com/oobabooga/one-click-installers

I also have alternative versions of it on my repo. I would recommend the Miniconda one if you are having issues with the normal installer: https://github.com/jllllll/one-click-installers/tree/oobabooga-windows-miniconda

johnswan commented 1 year ago

@johnswan https://github.com/oobabooga/one-click-installers

I also have alternative versions of it on my repo. I would recommend the Miniconda one if you are having issues with the normal installer: https://github.com/jllllll/one-click-installers/tree/oobabooga-windows-miniconda

Just ran that through a diff, the install.bat that I used last night is identical to the one in your repo:


@rem Based on the installer found here: https://github.com/Sygil-Dev/sygil-webui
@rem This script will install git and all dependencies
@rem using micromamba (an 8mb static-linked single-file binary, conda replacement).
@rem This enables a user to install this project without manually installing conda and git.

echo WARNING: This script relies on Micromamba which may have issues on some systems when installed under a path with spaces.
echo          May also have issues with long paths.&& echo.

echo What is your GPU?
echo.
echo A) NVIDIA
echo B) None (I want to run in CPU mode)
echo.
set /p "gpuchoice=Input> "
set gpuchoice=%gpuchoice:~0,1%

if /I "%gpuchoice%" == "A" (
    set "PACKAGES_TO_INSTALL=python=3.10.9 pytorch[version=2,build=py3.10_cuda11.7*] torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit ninja git"
    set "CHANNEL=-c pytorch -c nvidia/label/cuda-11.7.0 -c nvidia -c conda-forge"
) else if /I "%gpuchoice%" == "B" (
    set "PACKAGES_TO_INSTALL=pytorch torchvision torchaudio cpuonly git"
    set "CHANNEL=-c conda-forge -c pytorch"
) else (
    echo Invalid choice. Exiting...
    exit
)

cd /D "%~dp0"

set PATH=%PATH%;%SystemRoot%\system32

set MAMBA_ROOT_PREFIX=%cd%\installer_files\mamba
set INSTALL_ENV_DIR=%cd%\installer_files\env
set MICROMAMBA_DOWNLOAD_URL=https://github.com/mamba-org/micromamba-releases/releases/download/1.4.0-0/micromamba-win-64
set REPO_URL=https://github.com/oobabooga/text-generation-webui.git
set umamba_exists=F

@rem figure out whether git and conda needs to be installed
call "%MAMBA_ROOT_PREFIX%\micromamba.exe" --version >nul 2>&1
if "%ERRORLEVEL%" EQU "0" set umamba_exists=T

@rem (if necessary) install git and conda into a contained environment
if "%PACKAGES_TO_INSTALL%" NEQ "" (
    @rem download micromamba
    if "%umamba_exists%" == "F" (
        echo "Downloading Micromamba from %MICROMAMBA_DOWNLOAD_URL% to %MAMBA_ROOT_PREFIX%\micromamba.exe"

        mkdir "%MAMBA_ROOT_PREFIX%"
        call curl -Lk "%MICROMAMBA_DOWNLOAD_URL%" > "%MAMBA_ROOT_PREFIX%\micromamba.exe" || ( echo. && echo Micromamba failed to download. && goto end )

        @rem test the mamba binary
        echo Micromamba version:
        call "%MAMBA_ROOT_PREFIX%\micromamba.exe" --version || ( echo. && echo Micromamba not found. && goto end )
    )

    @rem create micromamba hook
    if not exist "%MAMBA_ROOT_PREFIX%\condabin\micromamba.bat" (
      call "%MAMBA_ROOT_PREFIX%\micromamba.exe" shell hook >nul 2>&1
    )

    @rem create the installer env
    if not exist "%INSTALL_ENV_DIR%" (
      echo Packages to install: %PACKAGES_TO_INSTALL%
      call "%MAMBA_ROOT_PREFIX%\micromamba.exe" create -y --prefix "%INSTALL_ENV_DIR%" %CHANNEL% %PACKAGES_TO_INSTALL% || ( echo. && echo Conda environment creation failed. && goto end )
    )
)

@rem check if conda environment was actually created
if not exist "%INSTALL_ENV_DIR%\python.exe" ( echo. && echo Conda environment is empty. && goto end )

@rem activate installer env
call "%MAMBA_ROOT_PREFIX%\condabin\micromamba.bat" activate "%INSTALL_ENV_DIR%" || ( echo. && echo MicroMamba hook not found. && goto end )

@rem clone the repository and install the pip requirements
if exist text-generation-webui\ (
  cd text-generation-webui
  git pull
) else (
  git clone https://github.com/oobabooga/text-generation-webui.git
  call python -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.37.2-py3-none-any.whl
  cd text-generation-webui || goto end
)
call python -m pip install -r requirements.txt --upgrade
call python -m pip install -r extensions\api\requirements.txt --upgrade
call python -m pip install -r extensions\elevenlabs_tts\requirements.txt --upgrade
call python -m pip install -r extensions\google_translate\requirements.txt --upgrade
call python -m pip install -r extensions\silero_tts\requirements.txt --upgrade
call python -m pip install -r extensions\whisper_stt\requirements.txt --upgrade

@rem skip gptq install if cpu only
if /I not "%gpuchoice%" == "A" goto bandaid

@rem download gptq and compile locally and if compile fails, install from wheel
if not exist repositories\ (
  mkdir repositories
)
cd repositories || goto end
if not exist GPTQ-for-LLaMa\ (
  git clone https://github.com/oobabooga/GPTQ-for-LLaMa.git -b cuda
  cd GPTQ-for-LLaMa || goto end
  call python -m pip install -r requirements.txt
  call python setup_cuda.py install
  if not exist "%INSTALL_ENV_DIR%\lib\site-packages\quant_cuda-0.0.0-py3.10-win-amd64.egg" (
    echo CUDA kernal compilation failed. Will try to install from wheel.
    call python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl || ( echo. && echo Wheel installation failed. && goto end )
  )
)

:end
pause
johnswan commented 1 year ago

@johnswan bitsandbytes does not support Windows. The installer uses a custom Windows-compatible version. If you were not using the latest installer, then you may not have gotten that version.

Wow... if bitsandbytes is not for Windows, then the installer is quite broken.

Giving up on this, I think aitrepeneur had a "special" version for his Youtube channel - or he faked it.

jllllll commented 1 year ago

@johnswan That is why I linked you to the Miniconda branch of my repo. Clone it with this command: git clone https://github.com/jllllll/one-click-installers.git -b oobabooga-windows-miniconda

Or download the files directly.

johnswan commented 1 year ago

Thanks, I tried but I got a bunch of errors at the install, first (extremely long) one seemed to be complaining about a missing DLL:


libcublas-dev-11.10. | 286.3 MB  | ##########################################################6                   |  76%
libcublas-dev-11.10. | 286.3 MB  | ##########################################################7                   |  76%
libcublas-dev-11.10. | 286.3 MB  | ##########################################################9                   |  77%
mpmath-1.3.0         | 428 KB    | ############################################################################# | 100%

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate C:\Users\John\Downloads\ow3\installer_files\env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Cloning into 'text-generation-webui'...
remote: Enumerating objects: 5254, done.
remote: Counting objects: 100% (1193/1193), done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 5254 (delta 1168), reused 1150 (delta 1150), pack-reused 4061
Receiving objects: 100% (5254/5254), 2.00 MiB | 6.82 MiB/s, done.
Resolving deltas: 100% (3478/3478), done.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Traceback (most recent call last):
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\misc.py", line 43, in <module>
    from pip._internal.locations import get_major_minor_version
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\__init__.py", line 14, in <module>
    from . import _sysconfig
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\locations\base.py", line 9, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 44, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 23, in _set_platform_dir_class
    from pip._vendor.platformdirs.windows import Windows as Result
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\Users\John\Downloads\ow3\installer_files\env\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
Cloning into 'GPTQ-for-LLaMa'...
remote: Enumerating objects: 811, done.
remote: Counting objects: 100% (811/811), done.```
Jeetkarsh commented 1 year ago

Screenshot 2023-04-09 at 11 47 56 PM Same issue with me, model loaded all well on terminal but nothing on GUI

jllllll commented 1 year ago

@johnswan Ok, there is definitely something wrong with your install if it can't even run pip. The _ctypes module that those errors are referring to is located in \installer_files\env\DLLs. Make sure that folder exists and isn't empty. If _ctypes is in there, then there is something wrong with CMD or your system. I am currently working on a new powershell version of the installer that should be more reliable than CMD. Here is an early version that seems to work on my system: powershell-one-click-installer.zip

@Jeetkarsh In your case, you can see CUDA extension not installed. in that log. You need GPTQ-for-LLaMa and to make sure to compile and install it's CUDA kernels. The kernels are what that error is referring to.

bbecausereasonss commented 1 year ago

Hmmm, this python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl --force-reinstall

has not fixed the recent errors loading some models >

johnswan commented 1 year ago

@jllllll

Thanks, but I'm not spending any more time on this.

Marfnl commented 1 year ago

I Had the same error but here is what works for me: call python server.py --chat --auto-devices --wbits 4 --groupsize 128

Disorbs commented 1 year ago

cd ".\text-generation-webui\repositories" rd /q /s ".\GPTQ-for-LLaMa\" >nul git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa.git -b cuda cd GPTQ-for-LLaMa git revert -n b55bd46b602ad497e0cbfe2ec2c7cce80013cebd python -m pip install -r requirements.txt python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/610fdae6588c2b17bcf2726cacaaf795cd45077e/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl --force-reinstall

**i tried this since i keep getting cuda extension not installed and all looked good but at the end it still says i dont have the extension installed?

and when i try to do it normally via python setup_cuda.py install i get this error - error pastebin**

Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:14:58) [MSC v.1929 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19045-SP0
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070 SUPER
Nvidia driver version: 531.41
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] clip-anytorch==2.5.0
[pip3] lion-pytorch==0.0.6
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] pytorch-lightning==1.9.0
[pip3] rotary-embedding-torch==0.2.1
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1
[pip3] torchdiffeq==0.2.3
[pip3] torchlibrosa==0.0.7
[pip3] torchmetrics==0.11.1
[pip3] torchsde==0.2.5
[pip3] torchvision==0.15.0
[conda] Could not collect
jllllll commented 1 year ago

@Disorbs Your Torch version is way out of date. You need Torch 2.0.0 and you have 1.13.1. As far as I'm aware, GPTQ-for-LLaMa is made using 2.0.0.

Disorbs commented 1 year ago

added 2.0.0 and re-did it looks good again but still the same cuda extension not found. Successfully installed quant-cuda-0.0.0

Death-777 commented 1 year ago

I'm not sure this will actually help any of you with your problem but I was getting this error before with Vicuna and GPT4xAlpaca-13b too and I set my virtual memory to "System managed size" for all my drives and it could finally load them. I hope it helps because I was really struggling to get any quantized models to load before that change, no matter what I did.

CyrisXD commented 1 year ago

@Death-777 This helped, thank you!

I've managed to get it working. In my case I had the model and files loading from my D: drive where it's memory management was set to 'none'.

I followed this tutorial on how to set the Windows PageFile size on my D: drive. I set it to System Managed Size and now the model loads correctly.

jashjasani commented 1 year ago

I tried this ->

Windows + R and search this "SystemPropertiesAdvanced.exe" without ", In the Performance tab click Settings and go to Advance In the Virtual Memory tab click Change, and turn off the "Automatically manage paging file size for all drives" Set all your Drive to System Managed Size click Set then Restart your device. I tried this and it worked

but now I'm getting a different error : _pickle.unpicklingerror: invalid load key, 'v'.

github-actions[bot] commented 11 months ago

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.