RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

[GeorvityLabs]

Describe the bug

RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions. 

@oobabooga @Loufe @luiscosio @Rudd-O @martinnj @xNul @lxe please suggest fixes for the same. I'm inside a docker container having cuda 11.6

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

!python setup_cuda.py install

Screenshot

No response

Logs

running bdist_egg
running egg_info
writing quant_cuda.egg-info/PKG-INFO
writing dependency_links to quant_cuda.egg-info/dependency_links.txt
writing top-level names to quant_cuda.egg-info/top_level.txt
reading manifest file 'quant_cuda.egg-info/SOURCES.txt'
writing manifest file 'quant_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
Traceback (most recent call last):
  File "/home/jovyan/text-generation-webui/repositories/GPTQ-for-LLaMa/setup_cuda.py", line 4, in <module>
    setup(
  File "/opt/conda/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/install.py", line 80, in run
    self.do_egg_install()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/install.py", line 129, in do_egg_install
    self.run_command('bdist_egg')
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 164, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 150, in call_command
    self.run_command(cmdname)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/command/install_lib.py", line 111, in build
    self.run_command('build_ext')
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/opt/conda/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
    self.build_extensions()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 499, in build_extensions
    _check_cuda_version(compiler_name, compiler_version)
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 386, in _check_cuda_version
    raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError: 
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.

System Info

Ubuntu 20.04

[GeorvityLabs]

![Uploading Screenshot from 2023-04-22 00-17-44.png…]()

[Loufe]

I'm not sure why I was tagged in this at all.

That said, are you sure you're using the most recent commit? I don't have any experience with Docker, but reading the Dockerfile setup I see only references to CUDA 11.8.

If you are, provide more details on what your setup looks like if you want help.

[jllllll]

https://github.com/oobabooga/one-click-installers/issues/29

Not sure how to do it in Docker, but you need to set CUDA_PATH="$PATH_TO_CUDA" to ensure that the correct cuda toolkit is selected when multiple are available.

[mrharrison007]

I got the same error using the one-click windows install. My Cuda I installed is 12.1, the only version that NVidia appears to have. Possibly the install recovered using wheel. I'll see if somehow I can verify GPU use if the app will run. After getting through the next part of the install.

_check_cuda_version(compiler_name, compiler_version)

File "C:\opt\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda)) RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

CUDA kernel compilation failed. Attempting installation with wheel. Collecting quant-cuda==0.0.0 Downloading https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 398.4/398.4 kB 3.6 MB/s eta 0:00:00 Installing collected packages: quant-cuda Successfully installed quant-cuda-0.0.0 Select the model that you want to download:

A) OPT 6.7B B) OPT 2.7B C) OPT 1.3B D) OPT 350M E) GALACTICA 6.7B F) GALACTICA 1.3B G) GALACTICA 125M H) Pythia-6.9B-deduped I) Pythia-2.8B-deduped J) Pythia-1.4B-deduped K) Pythia-410M-deduped L) Manually specify a Hugging Face model M) Do not download a model

Input>

[jllllll]

@gknerr Why did you manually install the latest version of CUDA Toolkit? The installer installs 11.7, which is what the webui uses.

Towards the bottom of start_windows.bat, just before call python webui.py, add this: set "CUDA_PATH=%INSTALL_ENV_DIR%"

For anyone else who has multiple versions of CUDA Toolkit, add that command to the start script, or if not on Windows, add the equivalent for your system.

[mrharrison007]

@jllllll I already had Cuda on my win10 workstation for some other pytorch work. You're a Rock Star, I'm up and running.

Gradio HTTP request redirected to localhost :) bin C:\opt\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll Loading anon8231489123_vicuna-13b-GPTQ-4bit-128g... Found the following quantized model: models\anon8231489123_vicuna-13b-GPTQ-4bit-128g\vicuna-13b-4bit-128g.safetensors Loading model ... Done. Loaded the model in 38.95 seconds. Loading the extension "gallery"... Ok. Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().

[xNul]

So it looks like we can just apply @jllllll's fix to the Windows one-click installer, but I wonder why it's happening in the first place. Once you're in the Miniconda environment, the CUDA installed to it should be used.

Maybe CUDA isn't being installed to the environment correctly because the command is being performed with Python's subprocess library which has weird interactions with environments. There may be other issues stemming from this too. I'll try to reproduce on my own machine and test

[jllllll]

@xNul It's being installed fine. It's just that there are multiple ways to locate the cuda toolkit that python packages use. You can either just search common file paths like bitsandbytes does, or you can use the CUDA_PATH environment variable to locate it. A lot of things, including compilers, use that variable. There are also variables for each of the cuda toolkit versions, though I haven't seen anything use them here.

This is really only an issue for people who have pre-installed toolkits as that variable will have already been set to one of those. I use that variable all the time for compiling cuda-related things to choose which toolkit to use. If everything just used PATH, then this wouldn't be such a problem.

[yunghoy]

download.py has the issue above. download was successful but the error message shows at the end.

executing the text-generation-webui has no issue.

[leblitzdick]

I also spent some time with it because the CUDA version 12.1 is also standard in my arch Linux(endeavouros) and there is no downgrade possibility to version 11.7/8, at least I have not found it!

Finally I tried the docker installation and that worked amazingly easy!

I would recommend this for all who use very current linux versions with CUDA 12.1 like arch. You save a lot of time and get to the target much faster.

[xbwtyz]

After using update_windows.bat my set up starting having this error. Reinstalling from the 1 click install resolves the issue but prohibits me from using the latest version.

[pawprint]

In case it helps someone - you can get the older CUDA toolkit here if you need it: https://developer.nvidia.com/cuda-11-7-0-download-archive

That said, I have been fighting with what I think are similar problems but this still isn't helping me. The one click install always fails to compile than says trying the Wheel but I never get anything but the following when it tries to run.

bin Z:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll Loading anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g... CUDA extension not installed.

This follows the use of the one-click installer and tried updating it. Have tried MANY completely fresh installs pip cache clears etc... would love any suggestions.

[jllllll]

@pawprint Try this installer: https://github.com/xNul/one-click-installers/archive/refs/heads/main.zip

It has some fixes that may help.

[pawprint]

Thanks @jllllll - tried that one with a fresh install but sadly same end result (although it's nice this one gives a message that the wheel install was a success)

RuntimeError: Error compiling objects for extension
ERROR: GPTQ CUDA kernel compilation failed.
Attempting installation with wheel.
Collecting quant-cuda==0.0.0
  Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Wheel installation success!
Continuing with install..

bin Z:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
Loading anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g...
CUDA extension not installed.
Found the following quantized model: models\anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g\gpt-x-alpaca-13b-native-4bit-128g-cuda.pt
Loading model ...
Done.
Loaded the model in 28.70 seconds.

and then when trying to interact it's the same

File "Z:\oobabooga\text-generation-webui\repositories\GPTQ-for-LLaMa\quant.py", line 426, in forward
    quant_cuda.vecquant4matmul(x, self.qweight, y, self.scales, self.qzeros, self.groupsize)
NameError: name 'quant_cuda' is not defined

I assume there is something "different" about my environment but I have no idea what it might be, automatic1111 and EveryDream training both are working fine.

[jllllll]

@pawprint Do you have conda globally installed and initialized on your system?

Try adding this to the top of the start script after @echo off:

SET CONDA_SHLVL=
SET PYTHONNOUSERSITE=1
SET PYTHONPATH=

If this does not help, open the cmd script and post the output of this command: conda info

[Tom-Neverwinter]

not going to get far without more hardware information... this is a very annoying trend. we need to know what you are compiling for.

[pawprint]

No change with that addition.

Yes I have conda (I use it to manage other environments) here is the info (slightly redacted)

(Z:\oobabooga\installer_files\env) Z:\oobabooga>conda info

     active environment : Z:\oobabooga\installer_files\env
    active env location : Z:\oobabooga\installer_files\env
            shell level : 1
       user config file : C:\Users\<username-redacted>\.condarc
 populated config files :
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.1=0
                          __win=0=0
       base environment : Z:\oobabooga\installer_files\conda  (writable)
      conda av data dir : Z:\oobabooga\installer_files\conda\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : Z:\oobabooga\installer_files\conda\pkgs
                          C:\Users\<username-redacted>\.conda\pkgs
                          C:\Users\<username-redacted>\AppData\Local\conda\conda\pkgs
       envs directories : Z:\oobabooga\installer_files\conda\envs
                          C:\Users\<username-redacted>\.conda\envs
                          C:\Users\<username-redacted>\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Windows/10 Windows/10.0.22621
          administrator : False
             netrc file : C:\Users\<username-redacted>/.netrc
           offline mode : False

Hardware (I had thought I included this but must have been another thread)

Core i7-8086K 64Gb RAM RTX 3090 24GB vRAM Windows 11 22621.1555

[harcont]

Thanks to @jllllll, I added CUDA_PATH=$INSTALL_ENV_DIR to the cmd_linux.sh script provided in the one click installer, then ran it and followed the guide, this fixed it for me.

@gknerr Why did you manually install the latest version of CUDA Toolkit? The installer installs 11.7, which is what the webui uses.

Towards the bottom of start_windows.bat, just before call python webui.py, add this: set "CUDA_PATH=%INSTALL_ENV_DIR%"

For anyone else who has multiple versions of CUDA Toolkit, add that command to the start script, or if not on Windows, add the equivalent for your system.

On windows it would be %INSTALL_ENV_DIR% instead of $INSTALL_ENV_DIR.

[xNul]

I just made some changes to the one-click installer here and I'm curious if they fix the issue without having to set the CUDA_PATH (the changes may fix the root cause of this problem). If someone could try it out, I'd appreciate it. If the error exists, we can just use the CUDA_PATH fix for all the one-click scripts

[akhil4rajan]

I also spent some time with it because the CUDA version 12.1 is also standard in my arch Linux(endeavouros) and there is no downgrade possibility to version 11.7/8, at least I have not found it!

Finally I tried the docker installation and that worked amazingly easy!

I would recommend this for all who use very current linux versions with CUDA 12.1 like arch. You save a lot of time and get to the target much faster.

How can we run it in docker. Any reference you can share ? @leblitzdick

[pawprint]

I'd appreciate it.

I'd try but I'm not super savvy with github so I'm not sure how to download a copy of the PR sufficient to try it.

[Tom-Neverwinter]

curious can it be set to pick the best fit automatically. 12.1 vs 11.8 vs 11.7 etc [I think I had 11.2 as tesla m40's had an issue with newer versions]

[xNul]

@pawprint thanks! You can download from my fork here https://github.com/xNul/one-click-installers. Or using this direct link https://github.com/xNul/one-click-installers/archive/refs/heads/main.zip

[xNul]

@Tom-Neverwinter each version of pytorch depends on a certain version of CUDA and text-generation-webui only has support for certain versions of pytorch. Adjusting the pytorch version to suit the already installed version of CUDA would be complicated and we'd run into a lot more bugs. Maybe in the future when things have settled down and text-generation-webui is focusing on maximum compatibility

[pawprint]

@xNul thanks for the attempt but it seems the same - I wiped out the folder and started from scratch. Install proceeded through to

RuntimeError: Error compiling objects for extension
ERROR: GPTQ CUDA kernel compilation failed.
Attempting installation with wheel.
Collecting quant-cuda==0.0.0
  Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Wheel installation success!
Continuing with install..

Which is the expected behavior I think?

but then when attempting to start up it's still:

Gradio HTTP request redirected to localhost :)
bin Z:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
Loading anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g...
CUDA extension not installed.
Found the following quantized model: models\anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g\gpt-x-alpaca-13b-native-4bit-128g-cuda.pt
Loading model ...
Done.

[xNul]

Does it give an error and exit? Very strange that it's still saying it can't find the CUDA extension after the wheel was installed correctly. Hopefully someone else figures out a fix because I can't reproduce. You might have better luck with another model though like https://huggingface.co/wcde/llama-7b-4bit-gr128

Does @jllllll's fix work for you?

[pawprint]

@xNul

Does @jllllll's fix work for you?

Sadly not. No difference.

No errors on the install it carries right through - the two snips I pasted were from the initial execution of the start script - it just kept on going. I tried a second time later opening the command prompt first to verify I was in fact in the correct env with it active before running start but same same.

I tried the other model - same deal.

Gradio HTTP request redirected to localhost :)
bin Z:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
The following models are available:

1. anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g
2. wcde_llama-7b-4bit-gr128

Which one do you want to load? 1-2

2

Loading wcde_llama-7b-4bit-gr128...
CUDA extension not installed.
Found the following quantized model: models\wcde_llama-7b-4bit-gr128\llama-7b-4bit-gr128.pt
Loading model ...
Done.

So, yes... I'm stumped as well - everything looks like it proceeds correctly - it's just not finding the CUDA extension. Out of curiosity - should I be able to find what is installed from the wheel somewhere? IE: is there a way to manually verify that the extention actually exists?

[xNul]

@pawprint it should be under Z:\oobabooga\installer_files\env\lib\site-packages and a folder called quant something .egg. You could also try opening the environment with cmd_windows.bat, running python with the python command, and importing the library. Another thing that might help is opening the environment and running the set command and copying the output here so I can look at your environment variables.

[pawprint]

@xNul Well that's interesting... there is nothing *.egg in that folder at all. But I do find quant_modules.py

No folders with quant_ in the name only 'quantize...' I wonder if it's not actually getting there. I could search more widely if I knew the exact name.

Perhaps of interest? in the Z:\oobabooga\installer_files\env folder

I find the CUDA_Toolkit_Release_Notes.txt which is for version 11.6 - but everything else seems to be 11.7 - could that be an issue?

Set output (some names redacted) :

(Z:\oobabooga\installer_files\env) Z:\oobabooga>set
ADSK_CLM_WPAD_PROXY_CHECK=FALSE
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\uname\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=EARTH
ComSpec=C:\WINDOWS\system32\cmd.exe
CONDA_DEFAULT_ENV=Z:\oobabooga\installer_files\env
CONDA_EXE=Z:\oobabooga\installer_files\conda\Scripts\conda.exe
CONDA_PREFIX=Z:\oobabooga\installer_files\env
CONDA_PROMPT_MODIFIER=(Z:\oobabooga\installer_files\env)
CONDA_PYTHON_EXE=Z:\oobabooga\installer_files\conda\python.exe
CONDA_ROOT_PREFIX=Z:\oobabooga\installer_files\conda
CONDA_SHLVL=1
DISTUTILS_USE_SDK=1
DokanLibrary1=C:\Program Files\Dokan\DokanLibrary-1.5.0\
DriverData=C:\Windows\System32\Drivers\DriverData
EFC_10752=1
FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer
FPS_BROWSER_USER_PROFILE_STRING=Default
HOMEDRIVE=C:
HOMEPATH=\Users\uname
INSTALL_ENV_DIR=Z:\oobabooga\installer_files\env
INTEL_DEV_REDIST=C:\Program Files (x86)\Common Files\Intel\Shared Libraries\
JAVA_HOME=C:\Program Files\Eclipse Adoptium\jdk-8.0.312.7-hotspot\
LOCALAPPDATA=C:\Users\uname\AppData\Local
LOGONSERVER=\\EARTH
MIC_LD_LIBRARY_PATH=C:\Program Files (x86)\Common Files\Intel\Shared Libraries\compiler\lib\mic
NUMBER_OF_PROCESSORS=12
NVTOOLSEXT_PATH=C:\Program Files\NVIDIA Corporation\NvToolsExt\
OneDrive=C:\Users\uname\OneDrive
OneDriveCommercial=C:\Users\uname\OneDrive
OPCODE6DIR64=C:\Program Files\Csound6_x64\plugins64\
OS=Windows_NT
Path=Z:\oobabooga\installer_files\env;Z:\oobabooga\installer_files\env\Library\mingw-w64\bin;Z:\oobabooga\installer_files\env\Library\usr\bin;Z:\oobabooga\installer_files\env\Library\bin;Z:\oobabooga\installer_files\env\Scripts;Z:\oobabooga\installer_files\env\bin;Z:\oobabooga\installer_files\conda\condabin;.;C:\Program Files\Eclipse Adoptium\jdk-8.0.312.7-hotspot\bin;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\Program Files (x86)\Razer\ChromaBroadcast\bin;C:\Program Files\Razer\ChromaBroadcast\bin;C:\Program Files (x86)\Razer Chroma SDK\bin;C:\Program Files\Razer Chroma SDK\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Users\uname\AppData\Local\Microsoft\WindowsApps;C:\Users\uname\AppData\Local\Programs\Python\Python38-32\Scripts;C:\Users\uname\AppData\Local\Programs\Python\Python38-32;C:\Program Files (x86)\GnuPG\bin;C:\Program Files\Csound6_x64\bin;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\dotnet;C:\Program Files\Git\cmd;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.2.0;C:\Users\uname\AppData\Local\Programs\Python\Python310\Scripts;C:\Users\uname\AppData\Local\Programs\Python\Python310;C:\Users\uname\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Intel\WiFi\bin;C:\Program Files\Common Files\Intel\WirelessCommon;C:\Users\uname\AppData\Local\Keybase;.;C:\WINDOWS\system32
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=9e0a
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=(Z:\oobabooga\installer_files\env) $P$G
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules
PUBLIC=C:\Users\Public
PYTHONPATH=C:\Program Files\Csound6_x64\bin\;
RAWWAVE_PATH=C:\Program Files\Csound6_x64\samples\
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\WINDOWS
TEMP=Z:\oobabooga\installer_files
TMP=Z:\oobabooga\installer_files
UGII_3DCONNEXION_DIR=C:\Program Files\3Dconnexion\3DxWare\3DxNX\ugii_dir\
UGII_3DCONNEXION_DIR32=C:\Program Files\3Dconnexion\3DxWare\3DxNX\ugii_dir32\
USERDOMAIN=EARTH
USERDOMAIN_ROAMINGPROFILE=EARTH
USERNAME=uname
USERPROFILE=C:\Users\uname
windir=C:\WINDOWS

[xNul]

@pawprint the full name of the folder is "quant_cuda-0.0.0-py3.10-win-amd64.egg". Try opening cmd_windows.bat and running this command python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl to install the wheel manually. Do you get any errors and does the folder show up afterwards?

[pawprint]

@xNul OK some progress... but....

No errors

(Z:\oobabooga\installer_files\env) Z:\oobabooga>python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl
Collecting quant-cuda==0.0.0
  Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Installing collected packages: quant-cuda
Successfully installed quant-cuda-0.0.0

Following that I end up with 1 folder: quant_cuda-0.0.0.dist-info but not the .egg one.

I also just noticed this file has appeared:

quant_cuda.cp310-win_amd64.pyd

Ok Tested it again - it's WORKING!!!

Thank you so much. That was the clincher. No idea why a manual install would change anything though. If you want me to try anything else that may help others I shall - but for now. It's working with both models.

[xNul]

@pawprint ayy awesome! Yeah I'm not sure either.

Could you replace the run_cmd function in webui.py with this code

def run_cmd(cmd, assert_success=False, environment=False, capture_output=False, env=None):
    # Use the conda environment
    if environment:
        conda_env_path = os.path.join(script_dir, "installer_files", "env")
        print(conda_env_path)
        if sys.platform.startswith("win"):
            cmd = "activate \"" + conda_env_path + "\" && " + cmd
        else:
            conda_sh_path = os.path.join(script_dir, "installer_files", "conda", "etc", "profile.d", "conda.sh")
            cmd = ". " + conda_sh_path + " && conda activate " + conda_env_path + " && " + cmd

    # Run shell commands
    result = subprocess.run(cmd, shell=True, capture_output=capture_output, env=env)

    # Assert the command ran successfully
    if assert_success and result.returncode != 0:
        print("Command '" + cmd + "' failed with exit status code '" + str(result.returncode) + "'. Exiting...")
        sys.exit()
    return result

delete the installer_files folder, delete the text-generation-webui folder, and try installing it again? I've made a couple changes to print out some more information which may show the bug if you could share the log again.

Also, is your Z drive a network drive?

[xNul]

ok actually, I was just able to reproduce this bug. Let's see what the problem is

[xNul]

@pawprint ok never mind. I wasn't able to reproduce. Could you still run the code I sent? The .egg folder is what you get when you compile everything correctly. The .pyd is what you get when you install from the wheel. I'm still not able to reproduce the scenario where both the wheel and the compile fail.

[pawprint]

@xNul Z is a local SSD.

I did the install in a new folder (had enough space - and saves me breaking what is working ;) )

ERROR: GPTQ CUDA kernel compilation failed.
Attempting installation with wheel.
Z:\ooba2\installer_files\env

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>conda.bat activate "Z:\ooba2\installer_files\env"
Collecting quant-cuda==0.0.0
  Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Wheel installation success!
Continuing with install..
Z:\ooba2\installer_files\env

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui>conda.bat activate "Z:\ooba2\installer_files\env"
Select the model that you want to download:

A) OPT 6.7B
B) OPT 2.7B
C) OPT 1.3B
D) OPT 350M
E) GALACTICA 6.7B
F) GALACTICA 1.3B
G) GALACTICA 125M
H) Pythia-6.9B-deduped
I) Pythia-2.8B-deduped
J) Pythia-1.4B-deduped
K) Pythia-410M-deduped
L) Manually specify a Hugging Face model
M) Do not download a model

Input> M
Z:\ooba2\installer_files\env

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui>conda.bat activate "Z:\ooba2\installer_files\env"
Gradio HTTP request redirected to localhost :)
bin Z:\ooba2\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
Loading wcde_llama-7b-4bit-gr128...
CUDA extension not installed.
Found the following quantized model: models\wcde_llama-7b-4bit-gr128\llama-7b-4bit-gr128.pt
Loading model ...
Done.

Here is the complete log: oobaInstall.txt

[xNul]

this confuses me further lol. Do you have the quant_modules.py file now too? Is there anything inside of it?

[pawprint]

I do - 30K file, it's in the folder: Z:\ooba2\installer_files\env\Lib\site-packages\transformers\models\ibert

[xNul]

ok thanks, that is something else. And there is no other file or folder starting with quant_cuda in the site-packages directory?

[pawprint]

nope - no folders nor files quant_* in site_packages

[xNul]

If you open up cmd_windows.bat and run the full command activate "Z:\ooba2\installer_files\env" && python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl, does it install the wheel correctly (i.e. does the quant_cuda-0.0.0.dist-info folder exist?)

[pawprint]

Baffling... Yes that worked. Got the folder and quant_cuda.cp310-win_amd64.pyd

((assuming that was conda activate....))

[xNul]

yeah it was and it's the exact command I use in the Python script to install the wheel. I'm trying to reproduce the bug while narrowing down where the problem is. Since installing the wheel works this way, this leads me to believe the bug has to do with the Python subprocess library which runs the commands. In the cmd_windows.bat please uninstall quant_cuda with pip uninstall quant_cuda, verify the folder and .pyd are gone, and then use the python command from cmd_windows.bat to open a Python interpreter and copy-paste this Python code

import os, sys, subprocess

script_dir = os.getcwd()

def run_cmd(cmd, assert_success=False, environment=False, capture_output=False, env=None):
    # Use the conda environment
    if environment:
        conda_env_path = os.path.join(script_dir, "installer_files", "env")
        if sys.platform.startswith("win"):
            cmd = "activate \"" + conda_env_path + "\" >nul && " + cmd
        else:
            conda_sh_path = os.path.join(script_dir, "installer_files", "conda", "etc", "profile.d", "conda.sh")
            cmd = ". " + conda_sh_path + " && conda activate " + conda_env_path + " && " + cmd

    # Run shell commands
    result = subprocess.run(cmd, shell=True, capture_output=capture_output, env=env)

    # Assert the command ran successfully
    if assert_success and result.returncode != 0:
        print("Command '" + cmd + "' failed with exit status code '" + str(result.returncode) + "'. Exiting...")
        sys.exit()
    return result

and then run

run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)

now see if quant_cuda-0.0.0.dist-info and quant_cuda.cp310-win_amd64.pyd exist or not

[xNul]

you could also put it into a Python file and run that with python file_here.py, whatever you prefer

[pawprint]

Did the first... and it installed correctly (which I know doesn't help narrow it down at all)

could it somehow not be in the right env in the full webui.py when it hits this point? I know it printed out that it was with that extra print you added.

[pawprint]

I just also did a pip list outside the env just to see if quant_cuda existed outside any env - but it doesn't.

[xNul]

idk honestly, this is so strange. We're activating the exact environment we installed with miniconda every time a command is run and the environment is also activated in the batch script that runs the python script itself. If you replace the webui.py line

result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)

with

result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=False)

and try installing again (deleting installer_files and text-generation-webui folders first), does it install the wheel correctly?

[pawprint]

I know enough python to see what you're doing at, yes it's very strange. I have the fresh install with the environment=false change running now... will let you know.

[xNul]

great thank you. I appreciate all the help. I have another idea which could possibly work, but I'm running out of them

[pawprint]

ok that did NOT install it either. It doesn't show up in pip list outside the env either (but I didn't go hunting for the file)

I should add the related lines:

ERROR: GPTQ CUDA kernel compilation failed.
Attempting installation with wheel.
Collecting quant-cuda==0.0.0
  Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Wheel installation success!
Continuing with install..
Z:\ooba2\installer_files\env

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui>conda.bat activate "Z:\ooba2\installer_files\env"
Select the model that you want to download:

Really happy to help if I can - would be nice to nail this down in case it hits others.

[xNul]

Let's try your idea. Maybe the correct environment isn't being activated because I don't activate the environment on Windows with the full path.

Try replacing this line

cmd = "activate \"" + conda_env_path + "\" && " + cmd

with these two lines

conda_bat_path = os.path.join(script_dir, "installer_files", "conda", "condabin", "conda.bat")
cmd = "\"" + conda_bat_path + "\" activate \"" + conda_env_path + "\" && " + cmd

and installing again