Closed SebJansen closed 9 months ago
turboderp
commented
11 months ago It does look like something breaks with CUDA 12.3. Quick search brings up a lot of results for that error, and apparently the workaround is something like:
export HOST_COMPILER=/usr/bin/g++-12I'll have to look into it later, maybe upgrade and see if I can reproduce it here.
picobyte
commented
11 months ago the exllamav2-0.0.7+cu118-cp310-cp310-linux_x86_64.whl installs cu12 EDIT: may have been resolved in the latest version.
lhl
commented
11 months ago This just bit me as well. Doing a search on the error message brought up this thread on nvcc. Here's the post from the Nvidia moderator:
The supported/tested gcc versions for any given CUDA version can be found in the CUDA linux install guide for that CUDA version. At the moment, here 6 (and here 10) is the one for 12.3, and you can see that gcc 13.x is not listed anywhere.
So there is no expectation by NVIDIA that CUDA 12.3 works with gcc 13.x. For CUDA 12.3, the stated gcc support goes up to gcc 12.2.
In my conda/mamba env, to get things working I installed:
mamba install gxx=12.2
oldgithubman
commented
6 months ago Having a similar problem:
Building wheels for collected packages: exllamav2
Building wheel for exllamav2 (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [50 lines of output]
Version: 0.0.17
warning: no previously-included files matching 'dni_*' found anywhere in distribution
/home/j/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:415: UserWarning: The detected CUDA version (12.4) has a minor version mismatch with the version that was used to compile PyTorch (12.1). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/home/j/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no /home/j/miniforge3/envs/exllamav2/bin/x86_64-conda-linux-gnu-c++ version bounds defined for CUDA version 12.4
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
Emitting ninja build file /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.11.1.git.kitware.jobserver-1
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cpp/quantize_func.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cpp/safetensors.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cpp/sampling.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/cache.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/kernel_select.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_1a.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_1b.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_2a.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_2b.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_3a.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_3b.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_gptq_1.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_gptq_2.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_gptq_3.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/h_add.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/h_gemm.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/layer_norm.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/lora.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/pack_tensor.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_attn.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_gemm.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_matrix.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_mlp.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/quantize.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/rms_norm.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/rope.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/util.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_bindings.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_cache.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_gemm.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_norm.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_qattn.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_qmatrix.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_qmlp.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_quant.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_rope.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_safetensors.o: No such file or directory
/home/j/miniforge3/envs/exllamav2/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_sampling.o: No such file or directory
collect2: error: ld returned 1 exit status
error: command '/home/j/miniforge3/envs/exllamav2/bin/x86_64-conda-linux-gnu-c++' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for exllamav2
Running setup.py clean for exllamav2
Failed to build exllamav2
ERROR: Could not build wheels for exllamav2, which is required to install pyproject.toml-based projects
siahuat0727
commented
6 months ago Having a similar problem:
Maybe try pip install --upgrade pip setuptools wheel before the installation. @oldmanjk
turboderp
commented
6 months ago It looks like it's not even trying to compile the sources and skipping straight to the linker even though there are no object files to link.
Try clearing out ~/.cache/torch_extensions perhaps? And also if you have a build directory in the repo dir, delete that.
oldgithubman
commented
6 months ago Having a similar problem:
Maybe try
pip install --upgrade pip setuptools wheelbefore the installation. @oldmanjk
Thanks for the suggestion, but nope.
~/exllamav2$ pip install --upgrade pip setuptools wheel
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /home/j/.local/lib/python3.10/site-packages (24.0)
Requirement already satisfied: setuptools in /home/j/.local/lib/python3.10/site-packages (69.2.0)
Requirement already satisfied: wheel in /home/j/.local/lib/python3.10/site-packages (0.43.0)It looks like it's not even trying to compile the sources and skipping straight to the linker even though there are no object files to link.
Try clearing out
~/.cache/torch_extensionsperhaps? And also if you have abuilddirectory in the repo dir, delete that.
Thanks for the suggestion, but nope. Cleared out ~/.cache/torch_extensions. No build directory.
~/exllamav2$ pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /home/j/exllamav2
Preparing metadata (setup.py) ... done
Requirement already satisfied: pandas in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (2.2.1)
Requirement already satisfied: ninja in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (1.11.1.1)
Requirement already satisfied: fastparquet in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (2024.2.0)
Requirement already satisfied: torch>=2.2.0 in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (2.2.1)
Requirement already satisfied: safetensors>=0.3.2 in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (0.4.2)
Requirement already satisfied: sentencepiece>=0.1.97 in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (0.1.99)
Requirement already satisfied: pygments in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (2.17.2)
Requirement already satisfied: websockets in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (11.0.3)
Requirement already satisfied: regex in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (2023.12.25)
Requirement already satisfied: numpy in /home/j/.local/lib/python3.10/site-packages (from exllamav2==0.0.17) (1.24.4)
Requirement already satisfied: filelock in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (3.13.3)
Requirement already satisfied: typing-extensions>=4.8.0 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (4.10.0)
Requirement already satisfied: sympy in /usr/lib/python3/dist-packages (from torch>=2.2.0->exllamav2==0.0.17) (1.9)
Requirement already satisfied: networkx in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (3.2.1)
Requirement already satisfied: jinja2 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (3.1.2)
Requirement already satisfied: fsspec in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (2024.2.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (2.19.3)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (12.1.105)
Requirement already satisfied: triton==2.2.0 in /home/j/.local/lib/python3.10/site-packages (from torch>=2.2.0->exllamav2==0.0.17) (2.2.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/j/.local/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=2.2.0->exllamav2==0.0.17) (12.4.99)
Requirement already satisfied: cramjam>=2.3 in /home/j/.local/lib/python3.10/site-packages (from fastparquet->exllamav2==0.0.17) (2.8.3)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from fastparquet->exllamav2==0.0.17) (23.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/j/.local/lib/python3.10/site-packages (from pandas->exllamav2==0.0.17) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3/dist-packages (from pandas->exllamav2==0.0.17) (2022.1)
Requirement already satisfied: tzdata>=2022.7 in /home/j/.local/lib/python3.10/site-packages (from pandas->exllamav2==0.0.17) (2024.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.2->pandas->exllamav2==0.0.17) (1.16.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/j/.local/lib/python3.10/site-packages (from jinja2->torch>=2.2.0->exllamav2==0.0.17) (2.1.5)
Building wheels for collected packages: exllamav2
Building wheel for exllamav2 (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [379 lines of output]
Version: 0.0.17
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching 'dni_*' found anywhere in distribution
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.exllamav2_ext' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.exllamav2_ext' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.exllamav2_ext' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.exllamav2_ext' to be distributed and are
already explicitly excluding 'exllamav2.exllamav2_ext' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.exllamav2_ext.cpp' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.exllamav2_ext.cpp' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.exllamav2_ext.cpp' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.exllamav2_ext.cpp' to be distributed and are
already explicitly excluding 'exllamav2.exllamav2_ext.cpp' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.exllamav2_ext.cuda' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.exllamav2_ext.cuda' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.exllamav2_ext.cuda' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.exllamav2_ext.cuda' to be distributed and are
already explicitly excluding 'exllamav2.exllamav2_ext.cuda' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.exllamav2_ext.cuda.comp_units' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.exllamav2_ext.cuda.comp_units' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.exllamav2_ext.cuda.comp_units' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.exllamav2_ext.cuda.comp_units' to be distributed and are
already explicitly excluding 'exllamav2.exllamav2_ext.cuda.comp_units' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.exllamav2_ext.cuda.quant' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.exllamav2_ext.cuda.quant' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.exllamav2_ext.cuda.quant' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.exllamav2_ext.cuda.quant' to be distributed and are
already explicitly excluding 'exllamav2.exllamav2_ext.cuda.quant' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.generator.filters' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.generator.filters' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.generator.filters' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.generator.filters' to be distributed and are
already explicitly excluding 'exllamav2.generator.filters' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.server' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.server' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.server' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.server' to be distributed and are
already explicitly excluding 'exllamav2.server' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'exllamav2.tokenizer' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'exllamav2.tokenizer' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'exllamav2.tokenizer' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'exllamav2.tokenizer' to be distributed and are
already explicitly excluding 'exllamav2.tokenizer' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
/home/j/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:415: UserWarning: The detected CUDA version (12.4) has a minor version mismatch with the version that was used to compile PyTorch (12.1). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/home/j/.local/lib/python3.10/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no x86_64-linux-gnu-g++ version bounds defined for CUDA version 12.4
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
Emitting ninja build file /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.11.1.git.kitware.jobserver-1
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cpp/quantize_func.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cpp/safetensors.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cpp/sampling.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/cache.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/kernel_select.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_1a.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_1b.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_2a.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_2b.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_3a.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_exl2_3b.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_gptq_1.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_gptq_2.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/comp_units/unit_gptq_3.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/h_add.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/h_gemm.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/layer_norm.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/lora.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/pack_tensor.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_attn.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_gemm.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_matrix.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/q_mlp.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/quantize.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/rms_norm.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/rope.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/cuda/util.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_bindings.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_cache.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_gemm.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_norm.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_qattn.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_qmatrix.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_qmlp.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_quant.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_rope.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_safetensors.o: No such file or directory
/usr/bin/ld: cannot find /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/exllamav2/exllamav2_ext/ext_sampling.o: No such file or directory
collect2: error: ld returned 1 exit status
error: command '/usr/bin/x86_64-linux-gnu-g++' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for exllamav2
Running setup.py clean for exllamav2
Failed to build exllamav2
ERROR: Could not build wheels for exllamav2, which is required to install pyproject.toml-based projectsIt's very odd. It's clearly attempting to compile even though it's not finding the compiler. But instead of stopping with an error message, it continues and tries to link the object that were never compiled. What do you get from gcc --version and nvcc --version?
oldgithubman
commented
6 months ago It's very odd. It's clearly attempting to compile even though it's not finding the compiler. But instead of stopping with an error message, it continues and tries to link the object that were never compiled. What do you get from
gcc --versionandnvcc --version?
$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0Are you sure there isn't a build directory? According to the output it writes the Ninja build file to /home/j/exllamav2/build/temp.linux-x86_64-cpython-310/build.ninja. If it is there, try clearing that whole directory, and if it still doesn't work, maybe post the build.ninja file?
oldgithubman
commented
6 months ago Are you sure there isn't a build directory? According to the output it writes the Ninja build file to
/home/j/exllamav2/build/temp.linux-x86_64-cpython-310/build.ninja. If it is there, try clearing that whole directory, and if it still doesn't work, maybe post the build.ninja file?
Yes, I'm sure. It creates a build directory during the process, but deletes it when it's done. It just magically started working though
Katehuuh
commented
1 month ago I have same error RuntimeError: Error compiling objects for extension by using "-allow-unsupported-compiler" after upgrade latest version of MSVC.
C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.41.34120/include\yvals_core.h(888): error: static assertion failed with "error STL1002: Unexpected compiler version, expected CUDA 12.4 or newer."
static_assert(false, "error " "STL1002" ": " "Unexpected compiler version, expected CUDA 12.4 or newer.");
^
Errors for no extra_cuda_cflags=: "-allow-unsupported-compiler" in setup.py return "Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler'...". However I already have VS 2022.
System info:
dev_tpnvcc --version: CUDA 12.1, V12.1.66python --version: 3.10.8cl --version: 19.41.34120 - Edit: Fix in VS, MSVC 14.34<14.40.33807 +PATH.gcc --version: 12.2.0 (MinGW-w64) turboderp
commented
1 month ago Microsoft broke compatibility at one point, and there's really nothing I can do about it from my end. You'll want to either update CUDA or downgrade Visual Studio.
JulienMaille
commented
1 month ago Microsoft broke compatibility at one point, and there's really nothing I can do about it from my end. You'll want to either update CUDA or downgrade Visual Studio.
This is also my conclusion but is it documented somewhere?
pip installing the JIT version fails so below I have pasted the error message for the build from source. I think a temporary workaround for me would be to grab a dockerfile that has CU11.8, because my rolling release distro updated cuda for me and I suspect that this is what made it stop working.My system:
Error message build from source
Errror message pip install JIT