RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

[GeorvityLabs]

Describe the bug

RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions. 

@oobabooga @Loufe @luiscosio @Rudd-O @martinnj @xNul @lxe please suggest fixes for the same. I'm inside a docker container having cuda 11.6

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

!python setup_cuda.py install

Screenshot

No response

Logs

running bdist_egg
running egg_info
writing quant_cuda.egg-info/PKG-INFO
writing dependency_links to quant_cuda.egg-info/dependency_links.txt
writing top-level names to quant_cuda.egg-info/top_level.txt
reading manifest file 'quant_cuda.egg-info/SOURCES.txt'
writing manifest file 'quant_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
Traceback (most recent call last):
  File "/home/jovyan/text-generation-webui/repositories/GPTQ-for-LLaMa/setup_cuda.py", line 4, in <module>
    setup(
  File "/opt/conda/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/install.py", line 80, in run
    self.do_egg_install()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/install.py", line 129, in do_egg_install
    self.run_command('bdist_egg')
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 164, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 150, in call_command
    self.run_command(cmdname)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/command/install_lib.py", line 111, in build
    self.run_command('build_ext')
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/dist.py", line 1244, in run_command
    super().run_command(command)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/opt/conda/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/opt/conda/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
    self.build_extensions()
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 499, in build_extensions
    _check_cuda_version(compiler_name, compiler_version)
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 386, in _check_cuda_version
    raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError: 
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.

System Info

Ubuntu 20.04

[xNul]

oh and change the environment=False back to environment=True please @pawprint

[pawprint]

Sorry to say... no dice. Output is identical but still no file or folder in site_packages.

[xNul]

if you run update_windows.bat does it install the wheel?

[pawprint]

I had tried that in the past it runs through trying but same result - it's not actually installed.

RuntimeError: Error compiling objects for extension
ERROR: GPTQ CUDA kernel compilation failed.
Attempting installation with wheel.
Z:\ooba2\installer_files\env
Collecting quant-cuda==0.0.0
  Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Wheel installation success!
Continuing with install..

Done!
Press any key to continue . . .

[xNul]

if you replace the line call python webui.py --update in update_windows.bat with cmd /k "python webui.py --update" does it work?

Also try typing where python using cmd_windows.bat and show what it outputs

[pawprint]

for the python:

(Z:\ooba2\installer_files\env) Z:\ooba2>where python
Z:\ooba2\installer_files\env\python.exe
C:\Users\uname\AppData\Local\Programs\Python\Python310\python.exe

trying the other now.

[pawprint]

and... no, still not showing up with the cmd /k "python webui.py --update" change.

[xNul]

ok please revert that change and open cmd_windows.bat again and run python webui.py --update from there. Does that work?

If it doesn't, we can start removing code from webui.py until it works to narrow down the cause.

[pawprint]

ok, the python webui.py --update from the cmd_windows.bat also didn't actually install it. Outputs are the same.

[xNul]

if you remove all code from the update_dependencies function in webui.py except for

# If the path does not exist, then the install failed
quant_cuda_path_regex = os.path.join(site_packages_path, "quant_cuda*/")
if not glob.glob(quant_cuda_path_regex):
    print("ERROR: GPTQ CUDA kernel compilation failed.")
    # Attempt installation via alternative, Windows-specific method
    if sys.platform.startswith("win"):
        print("Attempting installation with wheel.")
        result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)
        if result.returncode == 0:
            print("Wheel installation success!")
        else:
            print("ERROR: GPTQ wheel installation failed. You will not be able to run models in 4bit mode.")
    else:
        print("You will not be able to run models in 4bit mode.")
    print("Continuing with install..")

and then try to update, does it work?

[xNul]

oh wait, you have to leave this code in too

# Finds the path to your dependencies
for sitedir in site.getsitepackages():
    if "site-packages" in sitedir:
        site_packages_path = sitedir
        break

[pawprint]

YES - that worked

[xNul]

ok now we're getting somewhere lol

[xNul]

now use pip uninstall quant_cuda from cmd_windows.bat and revert the changes to update_dependencies. Instead of taking out all the code, let's only take half of it.

Try taking out everything between the

# Finds the path to your dependencies
for sitedir in site.getsitepackages():
    if "site-packages" in sitedir:
        site_packages_path = sitedir
        break

code and the

# If the path does not exist, then the install failed
quant_cuda_path_regex = os.path.join(site_packages_path, "quant_cuda*/")
if not glob.glob(quant_cuda_path_regex):
    print("ERROR: GPTQ CUDA kernel compilation failed.")
    # Attempt installation via alternative, Windows-specific method
    if sys.platform.startswith("win"):
        print("Attempting installation with wheel.")
        result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)
        if result.returncode == 0:
            print("Wheel installation success!")
        else:
            print("ERROR: GPTQ wheel installation failed. You will not be able to run models in 4bit mode.")
    else:
        print("You will not be able to run models in 4bit mode.")
    print("Continuing with install..")

code. Now see if it works

[pawprint]

Good... that also worked!

[xNul]

Awesome! Now let's uninstall quant_cuda with pip again, revert the changes again, and this time, I have a guess what's causing this. Try commenting out the line of code run_cmd("python setup_cuda.py install", environment=True) and see if it works.

[pawprint]

Hmmm nope that time it didn't actually install. Same output - just no file/folder

Could it have anything to do with the current dir at the time? (I know it shouldn't)

[xNul]

Could it have anything to do with the current dir at the time? (I know it shouldn't)

I've been wondering the same thing lol, but I think it's unlikely. I don't see how it would either

Ok if it's not the compilation that's causing the problem, then maybe it's this line run_cmd("python -m pip install -r requirements.txt", assert_success=True, environment=True). Try commenting that one out and see if it works

[pawprint]

That wasn't it either - same result - no file/folder.

[xNul]

ok how about we take out all this code this code

run_cmd("python -m pip install -r requirements.txt", assert_success=True, environment=True)

# On some Linux distributions, g++ may not exist or be the wrong version to compile GPTQ-for-LLaMa
if sys.platform.startswith("linux"):
    gxx_output = run_cmd("g++ --version", environment=True, capture_output=True)
    if gxx_output.returncode != 0 or b"g++ (GCC) 12" in gxx_output.stdout:
        # Install the correct version of g++
        run_cmd("conda install -y -k gxx_linux-64=11.2.0", environment=True)

run_cmd("python setup_cuda.py install", environment=True)

and see what happens to make sure. If that doesn't work, then let's try taking out

run_cmd("git pull", assert_success=True, environment=True)

[pawprint]

One sec... while I was waiting I tried something. With all code intact I just added

    os.chdir("..")
    os.chdir("..")

on the line before

# If the path does not exist, then the install failed

And it worked.

[xNul]

oh I have a guess what's happening. Print out the variable site_packages_path for me right before

# If the path does not exist, then the install failed

[pawprint]

Want me to keep the chdir's in or leave those out for this?

[xNul]

either way is good. The variable shouldn't be affected

[pawprint]

Result is

Z:\ooba2\installer_files\env\lib\site-packages

I should note that the actual path is

Z:\ooba2\installer_files\env\Lib\site-packages <-- note the uppercase L in Lib

[xNul]

ok actually there's no problem here. Let's try to make a minimal example. Let's remove all the code except for

# If the path does not exist, then the install failed
quant_cuda_path_regex = os.path.join(site_packages_path, "quant_cuda*/")
if not glob.glob(quant_cuda_path_regex):
    print("ERROR: GPTQ CUDA kernel compilation failed.")
    # Attempt installation via alternative, Windows-specific method
    if sys.platform.startswith("win"):
        print("Attempting installation with wheel.")
        result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)
        if result.returncode == 0:
            print("Wheel installation success!")
        else:
            print("ERROR: GPTQ wheel installation failed. You will not be able to run models in 4bit mode.")
    else:
        print("You will not be able to run models in 4bit mode.")
    print("Continuing with install..")

and

# Finds the path to your dependencies
for sitedir in site.getsitepackages():
    if "site-packages" in sitedir:
        site_packages_path = sitedir
        break

and leave all the os.chdir. Let's try to update and make sure it's not working, then add in those two os.chdir('..') you mentioned before and verify it is working.

[pawprint]

Just sanity check though - when we did the above without the chdir it did work. so I would expect it to again.

[xNul]

you mean when you added the two os.chdir('..'), it installed the wheel correctly? Yeah that's what we want to verify with the minimal example

[pawprint]

I mean the minimal example works without the os.chdir('..') - that was the same as the first one we tried. It only stops working when we start adding in more of the code back between those sections.

I expect it works because the working directory never changes - all those changes occur in the code between those 2 sections.

[xNul]

oh I meant for you to leave all the os.chdir in. Both the os.chdir('..') and the other os.chdir. If the location of the current directory is the cause, then leaving the os.chdir in will not install the wheel correctly until you add the os.chdir('..'). If that makes sense

[pawprint]

Ahhh ok Gotcha - will do.

so I have this

    os.chdir("text-generation-webui")

    # Finds the path to your dependencies
    for sitedir in site.getsitepackages():
        if "site-packages" in sitedir:
            site_packages_path = sitedir
            break

    os.chdir("repositories")

    os.chdir("GPTQ-for-LLaMa")

    #os.chdir("..")
    #os.chdir("..")
    #print(site_packages_path)
    # If the path does not exist, then the install failed
    quant_cuda_path_regex = os.path.join(site_packages_path, "quant_cuda*/")
    if not glob.glob(quant_cuda_path_regex):
        print("ERROR: GPTQ CUDA kernel compilation failed.")
        # Attempt installation via alternative, Windows-specific method
        if sys.platform.startswith("win"):
            print("Attempting installation with wheel.")
            result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)
            if result.returncode == 0:
                print("Wheel installation success!")
            else:
                print("ERROR: GPTQ wheel installation failed. You will not be able to use GPTQ-based models.")
        else:
            print("You will not be able to use GPTQ-based models.")
        print("Continuing with install..")

[pawprint]

Verified - it's NOT working without chdir up 2 and IS working with them back in.

[xNul]

perfect! Let's try to reduce it even further. Let's remove

# Finds the path to your dependencies
for sitedir in site.getsitepackages():
    if "site-packages" in sitedir:
        site_packages_path = sitedir
        break

and replace

# If the path does not exist, then the install failed
quant_cuda_path_regex = os.path.join(site_packages_path, "quant_cuda*/")
if not glob.glob(quant_cuda_path_regex):
    print("ERROR: GPTQ CUDA kernel compilation failed.")
    # Attempt installation via alternative, Windows-specific method
    if sys.platform.startswith("win"):
        print("Attempting installation with wheel.")
        result = run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)
        if result.returncode == 0:
            print("Wheel installation success!")
        else:
            print("ERROR: GPTQ wheel installation failed. You will not be able to run models in 4bit mode.")
    else:
        print("You will not be able to run models in 4bit mode.")
    print("Continuing with install..")

with

run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)

so that the update_dependencies function looks like this

    os.chdir("text-generation-webui")

    os.chdir("repositories")

    os.chdir("GPTQ-for-LLaMa")

    #os.chdir("..")
    #os.chdir("..")
    #print(site_packages_path)

    run_cmd("python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl", environment=True)

please verify it is not working without chdir up 2 and is working with them back in again.

[pawprint]

Yep - verified.

[xNul]

awesome, let's see if we can reproduce in cmd_window.bat now. Let's cd text-generation-webui\repositories\GPTQ-for-LLaMa and then try using python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl. Verify that does not install the wheel.

[pawprint]

I'm going to guess yes because I can't even UNINSTALL it from there but will still try.

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>pip uninstall quant_cuda
Found existing installation: quant-cuda 0.0.0
Can't uninstall 'quant-cuda'. No files were found to uninstall.

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>cd ..

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories>cd ..

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui>cd ..

(Z:\ooba2\installer_files\env) Z:\ooba2>pip uninstall quant_cuda
Found existing installation: quant-cuda 0.0.0
Uninstalling quant-cuda-0.0.0:
  Would remove:
    z:\ooba2\installer_files\env\lib\site-packages\quant_cuda-0.0.0.dist-info\*
    z:\ooba2\installer_files\env\lib\site-packages\quant_cuda.cp310-win_amd64.pyd
Proceed (Y/n)?

[pawprint]

Yes verified - it won't install down in that folder. No errors or anything - just doesn't happen.

[xNul]

try doing where pip and where python from the GPTQ-for-LLaMa folder

[pawprint]

as expected

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>where pip
Z:\ooba2\installer_files\env\Scripts\pip.exe
C:\Users\uname\AppData\Local\Programs\Python\Python310\Scripts\pip.exe

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>where python
Z:\ooba2\installer_files\env\python.exe
C:\Users\uname\AppData\Local\Programs\Python\Python310\python.exe

(Z:\ooba2\installer_files\env) Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>

[xNul]

if you open up a normal Command Prompt and cd to the GPTQ-for-LLaMa folder, are you still unable to install the wheel?

[pawprint]

Ahhhh... pip list there and I see quant_cuda

But no I can't uninstall it.

Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>pip list
Package            Version
------------------ ---------
aiofiles           22.1.0
aiohttp            3.8.3
aiosignal          1.3.1
altair             4.2.0
anyio              3.6.2
async-timeout      4.0.2
attrs              22.2.0
certifi            2022.12.7
charset-normalizer 2.1.1
click              8.1.3
colorama           0.4.6
contourpy          1.0.7
cycler             0.11.0
entrypoints        0.4
fastapi            0.89.1
ffmpy              0.3.0
fonttools          4.38.0
frozenlist         1.3.3
fsspec             2022.11.0
gradio             3.16.2
h11                0.14.0
httpcore           0.16.3
httpx              0.23.3
idna               3.4
Jinja2             3.1.2
jsonschema         4.17.3
kiwisolver         1.4.4
linkify-it-py      1.0.3
markdown-it-py     2.1.0
MarkupSafe         2.1.2
matplotlib         3.6.3
mdit-py-plugins    0.3.3
mdurl              0.1.2
multidict          6.0.4
ninja              1.11.1
numpy              1.24.1
opencv-python      4.7.0.72
orjson             3.8.5
packaging          23.0
pandas             1.5.2
Pillow             9.4.0
pip                22.2.2
pycryptodome       3.16.0
pydantic           1.10.4
pydub              0.25.1
pyparsing          3.0.9
pyrsistent         0.19.3
python-dateutil    2.8.2
python-multipart   0.0.5
pytz               2022.7.1
PyYAML             6.0
quant-cuda         0.0.0
requests           2.28.2
rfc3986            1.5.0
setuptools         63.2.0
six                1.16.0
sniffio            1.3.0
starlette          0.22.0
toolz              0.12.0
typing_extensions  4.4.0
uc-micro-py        1.0.1
urllib3            1.26.14
uvicorn            0.20.0
websockets         10.4
yarl               1.8.2

[notice] A new release of pip available: 22.2.2 -> 23.1.2
[notice] To update, run: python.exe -m pip install --upgrade pip

Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>pip uninstall quant_cuda
Found existing installation: quant-cuda 0.0.0
Can't uninstall 'quant-cuda'. No files were found to uninstall.

Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>

[xNul]

try doing where pip now. Is it the same?

[pawprint]

no in the normal prompt (no env) it's just my user one:

Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>where pip
C:\Users\uname\AppData\Local\Programs\Python\Python310\Scripts\pip.exe

[xNul]

if you go up a couple directories and use where pip does anything change?

[pawprint]

where pip does not change however... pip list does ... quant_cuda is gone when I'm in Z:\ooba2\text-generation-webui>

[xNul]

that is so strange lol. I've never run into anything like this before. Go back to the GPTQ-for-LLaMa folder and use the dir command please so I can see the directory.

[pawprint]

Ya I have never seen pip behaviour like this either...

 Directory of Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa

2023-04-27  19:57    <DIR>          .
2023-04-27  19:56    <DIR>          ..
2023-04-27  19:57    <DIR>          build
2023-04-27  19:56             1,143 convert_llama_weights_to_hf.py
2023-04-27  19:56             6,455 datautils.py
2023-04-27  19:56             5,622 gptq.py
2023-04-27  19:56            17,008 llama.py
2023-04-27  19:56             3,914 llama_inference.py
2023-04-27  19:56            13,368 llama_inference_offload.py
2023-04-27  19:56               413 modelutils.py
2023-04-27  19:56            19,058 opt.py
2023-04-27  19:56            20,184 quant.py
2023-04-27  19:56             4,260 quant_cuda.cpp
2023-04-27  20:58    <DIR>          quant_cuda.egg-info
2023-04-27  19:56            28,275 quant_cuda_kernel.cu
2023-04-27  19:56             9,088 README.md
2023-04-27  19:56                95 requirements.txt
2023-04-27  19:56               297 setup_cuda.py
2023-04-27  19:56             5,469 test_kernel.py
              15 File(s)        134,649 bytes
               4 Dir(s)  579,073,568,768 bytes free

Z:\ooba2\text-generation-webui\repositories\GPTQ-for-LLaMa>

[xNul]

this is what my directory looks like and I've confirmed pip list does not show quant-cuda for me like it does for you. Which is consistent with the fact that I don't have this bug.

04/27/2023  10:40 PM    <DIR>          .
04/27/2023  10:39 PM    <DIR>          ..
04/27/2023  10:39 PM             1,110 convert_llama_weights_to_hf.py
04/27/2023  10:39 PM             6,280 datautils.py
04/27/2023  10:39 PM             5,450 gptq.py
04/27/2023  10:39 PM            16,519 llama.py
04/27/2023  10:39 PM             3,784 llama_inference.py
04/27/2023  10:39 PM            13,368 llama_inference_offload.py
04/27/2023  10:39 PM               397 modelutils.py
04/27/2023  10:39 PM            18,554 opt.py
04/27/2023  10:39 PM            19,738 quant.py
04/27/2023  10:39 PM             4,142 quant_cuda.cpp
04/27/2023  10:39 PM            27,359 quant_cuda_kernel.cu
04/27/2023  10:39 PM             8,947 README.md
04/27/2023  10:39 PM                90 requirements.txt
04/27/2023  10:39 PM               287 setup_cuda.py
04/27/2023  10:39 PM             5,282 test_kernel.py
04/27/2023  10:40 PM    <DIR>          __pycache__
              15 File(s)        131,307 bytes
               3 Dir(s)  54,253,977,600 bytes free

This is what it looks like just from the clone and installing the wheel. It would probably look closer to yours if I tried to build. I'll do that rn so we can compare.

[pawprint]

I do have visual studio build tools on my system so it might be getting somewhere with the attempt at compiling that. Early on I had tried building it but gave up. obviously that was long before this current set of installs but the build tools are still there for other reasons.

[xNul]

This is what it looks like now after a successful compile/build

04/28/2023  12:48 AM    <DIR>          .
04/27/2023  10:39 PM    <DIR>          ..
04/28/2023  12:48 AM    <DIR>          build
04/27/2023  10:39 PM             1,110 convert_llama_weights_to_hf.py
04/27/2023  10:39 PM             6,280 datautils.py
04/28/2023  12:48 AM    <DIR>          dist
04/27/2023  10:39 PM             5,450 gptq.py
04/27/2023  10:39 PM            16,519 llama.py
04/27/2023  10:39 PM             3,784 llama_inference.py
04/27/2023  10:39 PM            13,368 llama_inference_offload.py
04/27/2023  10:39 PM               397 modelutils.py
04/27/2023  10:39 PM            18,554 opt.py
04/27/2023  10:39 PM            19,738 quant.py
04/27/2023  10:39 PM             4,142 quant_cuda.cpp
04/28/2023  12:48 AM    <DIR>          quant_cuda.egg-info
04/27/2023  10:39 PM            27,359 quant_cuda_kernel.cu
04/27/2023  10:39 PM             8,947 README.md
04/27/2023  10:39 PM                90 requirements.txt
04/27/2023  10:39 PM               287 setup_cuda.py
04/27/2023  10:39 PM             5,282 test_kernel.py
04/27/2023  10:40 PM    <DIR>          __pycache__
              15 File(s)        131,307 bytes
               6 Dir(s)  54,189,314,048 bytes free

pip list in a normal prompt still isn't showing quant-cuda