CUDA extension not installed.

[AiFLYF]

请问一下报这个错怎么解决，我系统上是有CUDA的，pytorch也检查的到，请问一下怎么解决？😃

[AiFLYF]

(qwen22) root@ed0d52e5239a:/home/Qwen2/examples/demo# pip list Package Version

---

accelerate 0.31.0 accessible-pygments 0.0.5 aiofiles 23.2.1 aiohttp 3.9.5 aiosignal 1.3.1 alabaster 0.7.16 altair 5.3.0 annotated-types 0.7.0 anyio 4.4.0 async-timeout 4.0.3 attrs 23.2.0 auto_gptq 0.7.1 Babel 2.15.0 beautifulsoup4 4.12.3 certifi 2024.6.2 charset-normalizer 3.3.2 click 8.1.7 coloredlogs 15.0.1 contourpy 1.2.1 cycler 0.12.1 datasets 2.19.2 dill 0.3.8 dnspython 2.6.1 docutils 0.21.2 email_validator 2.1.1 exceptiongroup 1.2.1 fastapi 0.111.0 fastapi-cli 0.0.4 ffmpy 0.3.2 filelock 3.14.0 fonttools 4.53.0 frozenlist 1.4.1 fsspec 2024.3.1 furo 2024.5.6 gekko 1.1.1 gradio 4.36.1 gradio_client 1.0.1 h11 0.14.0 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.23.3 humanfriendly 10.0 idna 3.7 imagesize 1.4.1 importlib_resources 6.4.0 Jinja2 3.1.4 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.9.0 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.5 multiprocess 0.70.16 networkx 3.2.1 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.1.105 nvidia-nvtx-cu12 12.1.105 optimum 1.20.0 orjson 3.10.4 packaging 24.1 pandas 2.2.2 peft 0.11.1 pillow 10.3.0 pip 24.0 protobuf 5.27.1 psutil 5.9.8 pyarrow 16.1.0 pyarrow-hotfix 0.6 pydantic 2.7.3 pydantic_core 2.18.4 pydata-sphinx-theme 0.15.3 pydub 0.25.1 Pygments 2.18.0 pyparsing 3.1.2 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-multipart 0.0.9 pytz 2024.1 PyYAML 6.0.1 referencing 0.35.1 regex 2024.5.15 requests 2.32.3 rich 13.7.1 rouge 1.0.1 rpds-py 0.18.1 ruff 0.4.8 safetensors 0.4.3 semantic-version 2.10.0 sentencepiece 0.2.0 setuptools 69.5.1 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 Sphinx 7.3.7 sphinx-basic-ng 1.0.0b2 sphinx-book-theme 1.1.2 sphinx-copybutton 0.5.2 sphinxcontrib-applehelp 1.0.8 sphinxcontrib-devhelp 1.0.6 sphinxcontrib-htmlhelp 2.0.5 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.7 sphinxcontrib-serializinghtml 1.1.10 starlette 0.37.2 sympy 1.12 tokenizers 0.19.1 tomli 2.0.1 tomlkit 0.12.0 toolz 0.12.1 torch 2.3.1+cu121 torchaudio 2.3.1+cu121 torchvision 0.18.1+cu121 tqdm 4.66.4 transformers 4.41.2 triton 2.3.1 typer 0.12.3 typing_extensions 4.12.2 tzdata 2024.1 ujson 5.10.0 urllib3 2.2.1 uvicorn 0.30.1 uvloop 0.19.0 watchfiles 0.22.0 websockets 11.0.3 wheel 0.43.0 xxhash 3.4.1 yarl 1.9.4

[jklj077]

Hi,

If you are using quantized models with GPTQ, you will need compatible versions of auto-gptq and torch. Please see https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/INSTALLATION.md for instruction.

P.S.: CUDA compiler (nvcc) is needed only if you need to install from the source and it should be of the same version as the CUDA for which torch is compiled. In many cases, you don't need to have it installed.