Cuda mismatch between installed and PyTorch causing AutoGPTQ error

[mcgeochd]

Hi All,

I had trouble getting ingest.py to run with dev or nightly versions of pytorch that support cuda 12.1, which I have installed:

(local-gpt) PS C:\Users\domin\Documents\Projects\Python\LocalGPT> nvidia-smi
Thu Jun 15 00:02:51 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98                 Driver Version: 535.98       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070 ...  WDDM  | 00000000:29:00.0  On |                  N/A |
|ERR!    0C    P0             ERR! / 235W |   1071MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

I was able to get it to run successfully with the following versions however, despite the cuda version mismatch:

# Name                    Version                   Build  Channel
torch                     2.0.1+cu118              pypi_0    pypi
torchvision               0.15.2+cu118             pypi_0    pypi

The problem now is that this version of pytorch is incompatible with AutoGPTQ, which I need as I don't have the vram to run a 7B model without quantisation. When I run pip install -r requirements.txt, I get the very long output:

(local-gpt) PS C:\Users\domin\Documents\Projects\Python\LocalGPT> pip install -r requirements.txt
Requirement already satisfied: langchain==0.0.191 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 2)) (0.0.191)
Requirement already satisfied: chromadb==0.3.22 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 3)) (0.3.22)        
Requirement already satisfied: llama-cpp-python==0.1.48 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 4)) (0.1.48)
Requirement already satisfied: pdfminer.six==20221105 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 5)) (20221105)
Requirement already satisfied: InstructorEmbedding in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 6)) (1.0.1)      
Requirement already satisfied: sentence-transformers in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 7)) (2.2.2)    
Requirement already satisfied: faiss-cpu in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 8)) (1.7.4)
Requirement already satisfied: huggingface_hub in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 9)) (0.15.1)
Requirement already satisfied: transformers in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 10)) (4.30.1)
Requirement already satisfied: protobuf==3.20.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 11)) (3.20.0)
Collecting auto-gptq
  Downloading auto_gptq-0.2.2.tar.gz (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.7/52.7 kB 674.6 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: urllib3==1.26.6 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 15)) (1.26.6)
Requirement already satisfied: accelerate in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 16)) (0.20.3)
Requirement already satisfied: bitsandbytes in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 17)) (0.39.0)
Requirement already satisfied: click in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 18)) (8.1.3)
Requirement already satisfied: openpyxl in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from -r requirements.txt (line 21)) (3.1.2)
Requirement already satisfied: PyYAML>=5.4.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (6.0)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (2.0.15)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (3.8.4)
Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (0.5.7)
Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (2.8.4)
Requirement already satisfied: numpy<2,>=1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (1.24.3)
Requirement already satisfied: openapi-schema-pydantic<2.0,>=1.2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (1.2.4)
Requirement already satisfied: pydantic<2,>=1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (1.10.9)
Requirement already satisfied: requests<3,>=2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (2.31.0)
Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from langchain==0.0.191->-r requirements.txt (line 2)) (8.2.2)
Requirement already satisfied: pandas>=1.3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (2.0.2)
Requirement already satisfied: hnswlib>=0.7 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.7.0)
Requirement already satisfied: clickhouse-connect>=0.5.7 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.6.2)
Requirement already satisfied: duckdb>=0.7.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.8.0)
Requirement already satisfied: fastapi>=0.85.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.96.0)
Requirement already satisfied: uvicorn[standard]>=0.18.3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (0.22.0)
Requirement already satisfied: posthog>=2.4.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (3.0.1)
Requirement already satisfied: typing-extensions>=4.5.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from chromadb==0.3.22->-r requirements.txt (line 3)) (4.6.3)
Requirement already satisfied: charset-normalizer>=2.0.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pdfminer.six==20221105->-r requirements.txt (line 5)) (3.1.0)
Requirement already satisfied: cryptography>=36.0.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pdfminer.six==20221105->-r requirements.txt (line 5)) (41.0.1)
Requirement already satisfied: tqdm in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (4.65.0)
Requirement already satisfied: torch>=1.6.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (2.0.1+cu118)
Requirement already satisfied: torchvision in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (0.15.2+cu118)
Requirement already satisfied: scikit-learn in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (1.2.2)
Requirement already satisfied: scipy in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (1.10.1)
Requirement already satisfied: nltk in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (3.8.1)
Requirement already satisfied: sentencepiece in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sentence-transformers->-r requirements.txt (line 7)) (0.1.99)
Requirement already satisfied: filelock in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from huggingface_hub->-r requirements.txt (line 9)) (3.12.1)
Requirement already satisfied: fsspec in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from huggingface_hub->-r requirements.txt (line 9)) (2023.6.0)
Requirement already satisfied: packaging>=20.9 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from huggingface_hub->-r requirements.txt (line 9)) (23.1)
Requirement already satisfied: regex!=2019.12.17 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from transformers->-r requirements.txt (line 10)) (2023.6.3)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from transformers->-r requirements.txt (line 10)) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from transformers->-r requirements.txt (line 10)) (0.3.1)
Collecting datasets
  Downloading datasets-2.13.0-py3-none-any.whl (485 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.6/485.6 kB 5.1 MB/s eta 0:00:00
Collecting rouge
  Downloading rouge-1.0.1-py3-none-any.whl (13 kB)
Requirement already satisfied: psutil in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from accelerate->-r requirements.txt (line 16)) (5.9.5)
Requirement already satisfied: colorama in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from click->-r requirements.txt (line 18)) (0.4.6)
Requirement already satisfied: et-xmlfile in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from openpyxl->-r requirements.txt (line 21)) (1.1.0)
Requirement already satisfied: attrs>=17.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (23.1.0)
Requirement already satisfied: multidict<7.0,>=4.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.191->-r requirements.txt (line 2)) (1.3.1)
Requirement already satisfied: certifi in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (2023.5.7)
Requirement already satisfied: pytz in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (2023.3)
Requirement already satisfied: zstandard in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (0.21.0)
Requirement already satisfied: lz4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from clickhouse-connect>=0.5.7->chromadb==0.3.22->-r requirements.txt (line 3)) (4.3.2)
Requirement already satisfied: cffi>=1.12 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from cryptography>=36.0.0->pdfminer.six==20221105->-r requirements.txt (line 5)) (1.15.1)
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (3.19.0)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (1.5.1)
Requirement already satisfied: typing-inspect>=0.4.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (0.9.0)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 3)) (0.27.0)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pandas>=1.3->chromadb==0.3.22->-r requirements.txt (line 3)) (2.8.2)
Requirement already satisfied: tzdata>=2022.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from pandas>=1.3->chromadb==0.3.22->-r requirements.txt (line 3)) (2023.3)
Requirement already satisfied: six>=1.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 3)) (1.16.0)
Requirement already satisfied: monotonic>=1.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 3)) (1.6)
Requirement already satisfied: backoff>=1.10.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from posthog>=2.4.0->chromadb==0.3.22->-r requirements.txt (line 3)) (2.2.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from requests<3,>=2->langchain==0.0.191->-r requirements.txt (line 2)) (3.4)
Requirement already satisfied: greenlet!=0.4.17 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from SQLAlchemy<3,>=1.4->langchain==0.0.191->-r requirements.txt (line 2)) (2.0.2)
Requirement already satisfied: sympy in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (1.12)
Requirement already satisfied: networkx in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (3.1)
Requirement already satisfied: jinja2 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (3.1.2)
Requirement already satisfied: h11>=0.8 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (0.14.0)
Requirement already satisfied: httptools>=0.5.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (0.5.0)
Requirement already satisfied: python-dotenv>=0.13 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (1.0.0)
Requirement already satisfied: watchfiles>=0.13 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (0.19.0)
Requirement already satisfied: websockets>=10.4 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb==0.3.22->-r requirements.txt (line 3)) (11.0.3)
Collecting pyarrow>=8.0.0
  Downloading pyarrow-12.0.1-cp311-cp311-win_amd64.whl (21.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.5/21.5 MB 15.6 MB/s eta 0:00:00
Collecting dill<0.3.7,>=0.3.0
  Downloading dill-0.3.6-py3-none-any.whl (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 kB ? eta 0:00:00
Collecting xxhash
  Downloading xxhash-3.2.0-cp311-cp311-win_amd64.whl (30 kB)
Collecting multiprocess
  Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.3/134.3 kB ? eta 0:00:00
Requirement already satisfied: joblib in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from nltk->sentence-transformers->-r requirements.txt (line 7)) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from scikit-learn->sentence-transformers->-r requirements.txt (line 7)) (3.1.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from torchvision->sentence-transformers->-r requirements.txt (line 7)) (9.5.0)
Requirement already satisfied: pycparser in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six==20221105->-r requirements.txt (line 5)) (2.21)
Requirement already satisfied: anyio<5,>=3.4.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 3)) (3.7.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from typing-inspect>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain==0.0.191->-r requirements.txt (line 2)) (1.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from jinja2->torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from sympy->torch>=1.6.0->sentence-transformers->-r requirements.txt (line 7)) (1.3.0)
Requirement already satisfied: sniffio>=1.1 in c:\users\domin\anaconda3\envs\local-gpt\lib\site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi>=0.85.1->chromadb==0.3.22->-r requirements.txt (line 3)) 
(1.3.0)
Building wheels for collected packages: auto-gptq
  Building wheel for auto-gptq (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [108 lines of output]
      include dirs are: ['autogptq_cuda']
      running bdist_wheel
      C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the 
slow distutils backend.
        warnings.warn(msg.format('we could not find ninja.'))
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-311
      creating build\lib.win-amd64-cpython-311\auto_gptq
      copying auto_gptq\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq
      creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\language_modeling_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\sequence_classification_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\text_summarization_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      creating build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\auto.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\bloom.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\codegen.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gpt2.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gptj.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gpt_bigcode.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gpt_neox.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\llama.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\moss.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\opt.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\rw.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\_const.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\fused_gptj_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\fused_llama_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\fused_llama_mlp.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\qlinear.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\qlinear_old.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\qlinear_triton.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\_fused_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      creating build\lib.win-amd64-cpython-311\auto_gptq\quantization
      copying auto_gptq\quantization\gptq.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
      copying auto_gptq\quantization\quantizer.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
      copying auto_gptq\quantization\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
      creating build\lib.win-amd64-cpython-311\auto_gptq\utils
      copying auto_gptq\utils\data_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
      copying auto_gptq\utils\import_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
      copying auto_gptq\utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
      creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      copying auto_gptq\eval_tasks\_utils\classification_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      copying auto_gptq\eval_tasks\_utils\generation_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      copying auto_gptq\eval_tasks\_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\custom_autotune.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\kernels.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\mixin.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      running build_ext
      C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
        warnings.warn(f'Error checking compiler version for {compiler}: {error}')
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\domin\AppData\Local\Temp\pip-install-ctgdafao\auto-gptq_81f9a71e890045dea95d7132c63017fe\setup.py", line 92, in <module>
          setup(
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
          super().run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\wheel\bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
          super().run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
          self.run_command(cmd_name)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
          super().run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run
          _build_ext.run(self)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
          self.build_extensions()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
          _check_cuda_version(compiler_name, compiler_version)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (12.1) mismatches the version that was used to compile
      PyTorch (11.8). Please make sure to use the same CUDA versions.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for auto-gptq
  Running setup.py clean for auto-gptq
Failed to build auto-gptq
Installing collected packages: xxhash, rouge, pyarrow, dill, multiprocess, datasets, auto-gptq
  Running setup.py install for auto-gptq ... error
  error: subprocess-exited-with-error

  × Running setup.py install for auto-gptq did not run successfully.
  │ exit code: 1
  ╰─> [124 lines of output]
      include dirs are: ['autogptq_cuda']
      running install
      C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
      !!

              ********************************************************************************
              Please avoid running ``setup.py`` directly.
              Instead, use pypa/build, pypa/installer, pypa/build or
              other standards-based tools.

              See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
              ********************************************************************************

      !!
        self.initialize_options()
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-311
      creating build\lib.win-amd64-cpython-311\auto_gptq
      copying auto_gptq\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq
      creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\language_modeling_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\sequence_classification_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\text_summarization_task.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      copying auto_gptq\eval_tasks\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks
      creating build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\auto.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\bloom.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\codegen.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gpt2.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gptj.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gpt_bigcode.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\gpt_neox.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\llama.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\moss.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\opt.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\rw.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\_const.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      copying auto_gptq\modeling\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\modeling
      creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\fused_gptj_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\fused_llama_attn.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\fused_llama_mlp.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\qlinear.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\qlinear_old.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\qlinear_triton.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\_fused_base.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      copying auto_gptq\nn_modules\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules
      creating build\lib.win-amd64-cpython-311\auto_gptq\quantization
      copying auto_gptq\quantization\gptq.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
      copying auto_gptq\quantization\quantizer.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
      copying auto_gptq\quantization\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\quantization
      creating build\lib.win-amd64-cpython-311\auto_gptq\utils
      copying auto_gptq\utils\data_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
      copying auto_gptq\utils\import_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
      copying auto_gptq\utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\utils
      creating build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      copying auto_gptq\eval_tasks\_utils\classification_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      copying auto_gptq\eval_tasks\_utils\generation_utils.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      copying auto_gptq\eval_tasks\_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\eval_tasks\_utils
      creating build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\custom_autotune.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\kernels.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\mixin.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      copying auto_gptq\nn_modules\triton_utils\__init__.py -> build\lib.win-amd64-cpython-311\auto_gptq\nn_modules\triton_utils
      running build_ext
      C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the 
slow distutils backend.
        warnings.warn(msg.format('we could not find ninja.'))
      C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
        warnings.warn(f'Error checking compiler version for {compiler}: {error}')
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\domin\AppData\Local\Temp\pip-install-ctgdafao\auto-gptq_81f9a71e890045dea95d7132c63017fe\setup.py", line 92, in <module>
          setup(
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
          super().run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\command\install.py", line 74, in run
          return orig.install.run(self)
                 ^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\install.py", line 697, in run
          self.run_command('build')
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
          super().run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build.py", line 131, in run
          self.run_command(cmd_name)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\dist.py", line 1244, in run_command
          super().run_command(command)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\command\build_ext.py", line 84, in run
          _build_ext.run(self)
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
          self.build_extensions()
        File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
          _check_cuda_version(compiler_name, compiler_version)
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (12.1) mismatches the version that was used to compile
      PyTorch (11.8). Please make sure to use the same CUDA versions.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> auto-gptq

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Until there is a stable version of pytorch for cuda 12.1, does anyone know how to fix this mismatch issue?

[LeafmanZ]

drop your nvidia driver version down to 515

[mcgeochd]

@LeafmanZ I went to https://www.nvidia.com/download/find.aspx to search for 515, but the oldest I found for my card, a 2070 super, was 527.56. Is there a version compatible with a 2070 super that will work?

[LeafmanZ]

Try it the driver and then see if nvidia-smi returns 11.8. I use ubuntu so idk tbh.

[mcgeochd]

Try it the driver and then see if nvidia-smi returns 11.8. I use ubuntu so idk tbh.

I can give it a go, but Table 3 in https://docs.nvidia.com/deploy/cuda-compatibility/index.html seems to suggest that 5.25+ only has compatibility with cuda 12.0 onwards. Though PyTorch compiled with 11.8 ran ingest.py despite my drivers being 12.1, so I’m not sure what’s going on. I do also have cudatoolkit 11.8 installed in the environment, but it doesn’t appear to have helped in this case.

[LeafmanZ]

yeah autoGPTQ is very very very picky. I was trying to run this on my windows (and WSL) set up for a while and just gave up and went back to ubuntu.

[LeafmanZ]

TBH its so confusing why windows is offered such a limited history of drivers, while linux u can go back over a year in driver history.

[CodeAKrome]

I fixed the error on ubuntu the following way and have submitted this text to README.md. No idea about windoze. conda create -n localGPT python=3.10 conda activate localGPT conda install -c nvidia cudatoolkit=11.7 conda init zsh

It's a million:1 odds, but it just might work ...

[teleprint-me]

The error in the output is saying that AutoGPTQ requires 11.8 which is probably the version number the dependency is set to in that project. We could, probably, include the required modules as submodules, but this would require local wheels to be built and compiling can take some time (sometimes more time than it's actually worth). Personally, I wouldn't bother. Downgrading is the best option here. You would need to look at the official nvidia driver listing and download the right one and then use DDU to swap the drivers out. It is an annoying and time consuming process. If you're on W11, you're most likely locked in as Microsoft now force distributes their drivers and driver updates. I know that DDU is still useful and still used, but I don't use Windows anymore (for years now) and it could be something that could break your system install along the way. I have no idea tbh.

[Jawn78]

You may have too new of a cuda for auto-gptq to build the wheel. You should install cu118 or cu117, depending on the version of Cuda you choose.

pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/