setup.py doesn't work - Githubissues

When I create conda environment using these steps:

conda create --name gguf-to-torch python=3.12 -y
conda activate gguf-to-torch

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia -y

CMAKE_ARGS="-DLLAMA_CUDA=on" pip install llama-cpp-python

pip install gguf sentencepiece

conda install nvidia/label/cuda-12.1.0::cuda -y

Then run

python setup.py install

It generates the following exception:

running bdist_egg
running egg_info
writing llamacpp_cuda.egg-info/PKG-INFO
writing dependency_links to llamacpp_cuda.egg-info/dependency_links.txt
writing top-level names to llamacpp_cuda.egg-info/top_level.txt
/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/utils/cpp_extension.py:500: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'llamacpp_cuda.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'llamacpp_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no g++ version bounds defined for CUDA version 12.1
  warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'llamacpp_cuda' extension
/home/zeio/.conda/envs/gguf-to-torch/bin/nvcc -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include/torch/csrc/api/include -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include/TH -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include/THC -I/home/zeio/.conda/envs/gguf-to-torch/include -I/home/zeio/.conda/envs/gguf-to-torch/include/python3.12 -c llamacpp_kernel.cu -o build/temp.linux-x86_64-cpython-312/llamacpp_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 -g -Xcompiler -rdynamic -lineinfo -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=llamacpp_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
gcc -pthread -B /home/zeio/.conda/envs/gguf-to-torch/compiler_compat -fno-strict-overflow -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /home/zeio/.conda/envs/gguf-to-torch/include -fPIC -O2 -isystem /home/zeio/.conda/envs/gguf-to-torch/include -fPIC -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include/torch/csrc/api/include -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include/TH -I/home/zeio/.conda/envs/gguf-to-torch/lib/python3.12/site-packages/torch/include/THC -I/home/zeio/.conda/envs/gguf-to-torch/include -I/home/zeio/.conda/envs/gguf-to-torch/include/python3.12 -c py_bind.cpp -o build/temp.linux-x86_64-cpython-312/py_bind.o -g -lineinfo -fno-strict-aliasing -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=llamacpp_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
py_bind.cpp:25:1: error: two or more data types in declaration of ‘ggml_mul_mat_a8’
   25 | torch::Tensor void ggml_mul_mat_a8(
      | ^~~~~
py_bind.cpp: In function ‘void pybind11_init_llamacpp_cuda(pybind11::module_&)’:
py_bind.cpp:36:31: error: ‘ggml_mul_mat_a8’ was not declared in this scope; did you mean ‘ggml_mul_mat_vec’?
   36 |     m.def("ggml_mul_mat_a8", &ggml_mul_mat_a8, "ggml_mul_mat_a8");
      |                               ^~~~~~~~~~~~~~~
      |                               ggml_mul_mat_vec
error: command '/usr/bin/gcc' failed with exit code 1

What might be wrong?

chu-tianxiang / llama-cpp-torch

setup.py doesn't work #1