error in gpu support

[dpenra]

system ubuntu,cpu ryzen 3600, GPU RTX 2060 super

CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python Collecting llama-cpp-python Downloading llama_cpp_python-0.2.26.tar.gz (8.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 5.6 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing metadata (pyproject.toml) ... done Collecting typing-extensions>=4.5.0 (from llama-cpp-python) Downloading typing_extensions-4.9.0-py3-none-any.whl.metadata (3.0 kB) Collecting numpy>=1.20.0 (from llama-cpp-python) Downloading numpy-1.26.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 17.2 MB/s eta 0:00:00 Collecting diskcache>=5.6.1 (from llama-cpp-python) Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB) Downloading diskcache-5.6.3-py3-none-any.whl (45 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 163.4 MB/s eta 0:00:00 Downloading numpy-1.26.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 10.5 MB/s eta 0:00:00 Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB) Building wheels for collected packages: llama-cpp-python Building wheel for llama-cpp-python (pyproject.toml) ... error error: subprocess-exited-with-error

× Building wheel for llama-cpp-python (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [39 lines of output] scikit-build-core 0.7.1 using CMake 3.22.1 (wheel) Configuring CMake... 2024-01-03 12:38:20,333 - scikit_build_core - WARNING - libdir/ldlibrary: /usr/lib/x86_64-linux-gnu/libpython3.11.so is not a real file! 2024-01-03 12:38:20,333 - scikit_build_core - WARNING - Can't find a Python library, got libdir=/usr/lib/x86_64-linux-gnu, ldlibrary=libpython3.11.so, multiarch=x86_64-linux-gnu, masd=x86_64-linux-gnu loading initial cache file /tmp/tmpw13hw4ga/build/CMakeInit.txt -- The C compiler identification is GNU 11.4.0 -- The CXX compiler identification is GNU 11.4.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found Git: /usr/bin/git (found version "2.34.1") -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE -- Found CUDAToolkit: /usr/local/cuda/include (found version "12.3.103") -- cuBLAS found -- The CUDA compiler identification is unknown CMake Error at vendor/llama.cpp/CMakeLists.txt:268 (enable_language): No CMAKE_CUDA_COMPILER could be found.

    Tell CMake where to find the compiler by setting either the environment
    variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full
    path to the compiler, or to the compiler name if it is in the PATH.

  -- Configuring incomplete, errors occurred!
  See also "/tmp/tmpw13hw4ga/build/CMakeFiles/CMakeOutput.log".
  See also "/tmp/tmpw13hw4ga/build/CMakeFiles/CMakeError.log".

  *** CMake configuration failed
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for llama-cpp-python Failed to build llama-cpp-python ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects

[AIrabbit0]

I have a similar error ```Building wheels for collected packages: llama-cpp-python Building wheel for llama-cpp-python (pyproject.toml) ... error error: subprocess-exited-with-error

× Building wheel for llama-cpp-python (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [40 lines of output] scikit-build-core 0.7.1 using CMake 3.28.1 (wheel) Configuring CMake... 2024-01-04 06:46:45,834 - scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None loading initial cache file C:\Users\PCG\AppData\Local\Temp\tmpy64sqe8o\build\CMakeInit.txt -- Building for: Visual Studio 17 2022 -- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631. -- The C compiler identification is MSVC 19.38.33133.0 -- The CXX compiler identification is MSVC 19.38.33133.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.43.0.windows.1") -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - not found -- Found Threads: TRUE -- Found CUDAToolkit: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.2/include (found version "12.2.91") -- cuBLAS found CMake Error at C:/Users/PCG/AppData/Local/Temp/pip-build-env-1mh4nb2g/normal/Lib/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:529 (message): No CUDA toolset found. Call Stack (most recent call first): C:/Users/PCG/AppData/Local/Temp/pip-build-env-1mh4nb2g/normal/Lib/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD) C:/Users/PCG/AppData/Local/Temp/pip-build-env-1mh4nb2g/normal/Lib/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test) C:/Users/PCG/AppData/Local/Temp/pip-build-env-1mh4nb2g/normal/Lib/site-packages/cmake/data/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:135 (CMAKE_DETERMINE_COMPILER_ID) vendor/llama.cpp/CMakeLists.txt:268 (enable_language)

  -- Configuring incomplete, errors occurred!

  *** CMake configuration failed
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for llama-cpp-python Failed to build llama-cpp-python ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects```

[hyperinx]

CMAKE_ARGS="-DLLAMA_CUBLAS=on -DCUDA_PATH=/usr/local/cuda-12.2 -DCUDAToolkit_ROOT=/usr/local/cuda-12.2 -DCUDAToolkit_INCLUDE_DIR=/usr/local/cuda-12/include -DCUDAToolkit_LIBRARY_DIR=/usr/local/cuda-12.2/lib64" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir --verbose

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.2/compat (fixes the error with missing libcuda.so file)

Change path of cuda-12.X to your version of cuda.

If you are using it with docker you should create an image from nvidia/cuda:12.2.2-devel-ubuntu22.04 for example.

In my case it works only with devel docker image Install on your host https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html You can run it with nvidia support by using this cmd: docker run --runtime=nvidia -p 8001:8001

Dockerfile example:

FROM nvidia/cuda:12.2.2-devel-ubuntu22.04

RUN apt update && apt install -y git gcc

RUN apt-get update -y \ && apt-get upgrade -y \ && apt-get -y install build-essential \ zlib1g-dev \ libncurses5-dev \ libgdbm-dev \ libnss3-dev \ libssl-dev \ libreadline-dev \ libffi-dev \ libsqlite3-dev \ libbz2-dev \ wget \ && export DEBIAN_FRONTEND=noninteractive \ && apt-get purge -y imagemagick imagemagick-6-common

RUN cd /usr/src \ && wget https://www.python.org/ftp/python/3.11.0/Python-3.11.0.tgz \ && tar -xzf Python-3.11.0.tgz \ && cd Python-3.11.0 \ && ./configure --enable-optimizations \ && make altinstall

RUN update-alternatives --install /usr/bin/python python /usr/local/bin/python3.11 1

RUN apt install -y git python3-pip gcc

RUN git clone https://github.com/imartinez/privateGPT.git /app WORKDIR /app

RUN pip install poetry

RUN poetry env use 3.11.0

RUN poetry add sentence-transformers RUN poetry add huggingface_hub RUN poetry install --with ui,local && poetry run ./scripts/setup

ENV CMAKE_ARGS="-DLLAMA_CUBLAS=on -DCUDA_PATH=/usr/local/cuda-12.2 -DCUDAToolkit_ROOT=/usr/local/cuda-12.2 -DCUDAToolkit_INCLUDE_DIR=/usr/local/cuda-12/include -DCUDAToolkit_LIBRARY_DIR=/usr/local/cuda-12.2/lib64" ENV FORCE_CMAKE=1

RUN pip install llama-cpp-python --no-cache-dir

RUN poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.2/compat

EXPOSE 8001

ENTRYPOINT ["make", "run"]

docker-compose: version: '3.5'

services: privategpt: build: . image: privateGPT volumes:

• ./data/db:/app/db
• ./data/models:/app/models
• ./data/source_documents:/app/source_documents
• ./data/torch:/root/.cache/torch deploy: resources: reservations: devices:
  • driver: nvidia count: 1 capabilities: [gpu]

[dpenra]

I found the solution. This is mainly because the system is unable to detect GPUs. Try to get info about your system gpu. By typing nvidia-smi and nvcc -V . If you cant access your gpu information you might have to install gpu driver in your system in order to use gpu accelerated privateGPT.

[AIrabbit0]

ERROR: Option and is not recognized. Please run 'nvidia-smi -h'. and I runned that line and I got Supported products:

• Full Support
  • All Tesla products, starting with the Kepler architecture
  • All Quadro products, starting with the Kepler architecture
  • All GRID products, starting with the Kepler architecture
  • GeForce Titan products, starting with the Kepler architecture

• Limited Support

  • All Geforce products, starting with the Kepler architecture nvidia-smi [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...

  -h, --help Print usage information and exit.

  LIST OPTIONS:

  -L, --list-gpus Display a list of GPUs connected to the system.

  -B, --list-excluded-gpus Display a list of excluded GPUs in the system.

  SUMMARY OPTIONS:

  Show a summary of GPUs connected to the system. [plus any of] -i, --id= Target a specific GPU. -f, --filename= Log to a specified file, rather than to stdout. -l, --loop= Probe until Ctrl+C at specified second interval. QUERY OPTIONS: -q, --query Display GPU or Unit info. [plus any of] -u, --unit Show unit, rather than GPU, attributes. -i, --id= Target a specific GPU or Unit. -f, --filename= Log to a specified file, rather than to stdout. -x, --xml-format Produce XML output. --dtd When showing xml output, embed DTD. -d, --display= Display only selected information: MEMORY, UTILIZATION, ECC, TEMPERATURE, POWER, CLOCK, COMPUTE, PIDS, PERFORMANCE, SUPPORTED_CLOCKS, PAGE_RETIREMENT, ACCOUNTING, ENCODER_STATS, SUPPORTED_GPU_TARGET_TEMP, VOLTAGE, FBC_STATS ROW_REMAPPER, RESET_STATUS Flags can be combined with comma e.g. ECC,POWER. Sampling data with max/min/avg is also returned for POWER, UTILIZATION and CLOCK display types. Doesn't work with -u or -x flags. -l, --loop= Probe until Ctrl+C at specified second interval. -lms, --loop-ms= Probe until Ctrl+C at specified millisecond interval. SELECTIVE QUERY OPTIONS: Allows the caller to pass an explicit list of properties to query. [one of] --query-gpu Information about GPU. Call --help-query-gpu for more info. --query-supported-clocks List of supported clocks. Call --help-query-supported-clocks for more info. --query-compute-apps List of currently active compute processes. Call --help-query-compute-apps for more info. --query-accounted-apps List of accounted compute processes. Call --help-query-accounted-apps for more info. This query is not supported on vGPU host. --query-retired-pages List of device memory pages that have been retired. Call --help-query-retired-pages for more info. --query-remapped-rows Information about remapped rows. Call --help-query-remapped-rows for more info. [mandatory] --format= Comma separated list of format options: csv - comma separated values (MANDATORY) noheader - skip the first line with column headers nounits - don't print units for numerical values [plus any of] -i, --id= Target a specific GPU or Unit. -f, --filename= Log to a specified file, rather than to stdout. -l, --loop= Probe until Ctrl+C at specified second interval. -lms, --loop-ms= Probe until Ctrl+C at specified millisecond interval. DEVICE MODIFICATION OPTIONS: [any one of] -e, --ecc-config= Toggle ECC support: 0/DISABLED, 1/ENABLED -p, --reset-ecc-errors= Reset ECC error counts: 0/VOLATILE, 1/AGGREGATE -c, --compute-mode= Set MODE for compute applications: 0/DEFAULT, 1/EXCLUSIVE_THREAD (DEPRECATED), 2/PROHIBITED, 3/EXCLUSIVE_PROCESS -dm, --driver-model= Enable or disable TCC mode: 0/WDDM, 1/TCC -fdm, --force-driver-model= Enable or disable TCC mode: 0/WDDM, 1/TCC Ignores the error that display is connected. --gom= Set GPU Operation Mode: 0/ALL_ON, 1/COMPUTE, 2/LOW_DP -lgc --lock-gpu-clocks= Specifies clocks as a pair (e.g. 1500,1500) that defines the range of desired locked GPU clock speed in MHz. Setting this will supercede application clocks and take effect regardless if an app is running. Input can also be a singular desired clock value (e.g. ). Optionally, --mode can be specified to indicate a special mode. -m --mode= Specifies the mode for --locked-gpu-clocks. Valid modes: 0, 1 -rgc --reset-gpu-clocks Resets the Gpu clocks to the default values. -lmc --lock-memory-clocks= Specifies clocks as a pair (e.g. 5100,5100) that defines the range of desired locked Memory clock speed in MHz. Input can also be a singular desired clock value (e.g. ). -rmc --reset-memory-clocks Resets the Memory clocks to the default values. -lmcd --lock-memory-clocks-deferred= Specifies memClock clock to lock. This limit is applied the next time GPU is initialized. This is guaranteed by unloading and reloading the kernel module. Requires root. -rmcd --reset-memory-clocks-deferred Resets the deferred Memory clocks applied. -ac --applications-clocks= Specifies clocks as a pair (e.g. 2000,800) that defines GPU's speed in MHz while running applications on a GPU. -rac --reset-applications-clocks Resets the applications clocks to the default values. -pl --power-limit= Specifies maximum power management limit in watts. Takes an optional argument --scope. -sc --scope= Specifies the device type for --scope: 0/GPU, 1/TOTAL_MODULE (Grace Hopper Only) -cc --cuda-clocks= Overrides or restores default CUDA clocks. In override mode, GPU clocks higher frequencies when running CUDA applications. Only on supported devices starting from the Volta series. Requires administrator privileges. 0/RESTORE_DEFAULT, 1/OVERRIDE -am --accounting-mode= Enable or disable Accounting Mode: 0/DISABLED, 1/ENABLED -caa --clear-accounted-apps Clears all the accounted PIDs in the buffer. --auto-boost-default= Set the default auto boost policy to 0/DISABLED or 1/ENABLED, enforcing the change only after the last boost client has exited. --auto-boost-permission= Allow non-admin/root control over auto boost mode: 0/UNRESTRICTED, 1/RESTRICTED -mig --multi-instance-gpu= Enable or disable Multi Instance GPU: 0/DISABLED, 1/ENABLED Requires root. -gtt --gpu-target-temp= Set GPU Target Temperature for a GPU in degree celsius. Requires administrator privileges [plus optional] -i, --id= Target a specific GPU. -eow, --error-on-warning Return a non-zero error for warnings. UNIT MODIFICATION OPTIONS: -t, --toggle-led= Set Unit LED state: 0/GREEN, 1/AMBER [plus optional] -i, --id= Target a specific Unit. SHOW DTD OPTIONS: --dtd Print device DTD and exit. [plus optional] -f, --filename= Log to a specified file, rather than to stdout. -u, --unit Show unit, rather than device, DTD. --debug= Log encrypted debug information to a specified file. Device Monitoring: dmon Displays device stats in scrolling format. "nvidia-smi dmon -h" for more information. daemon Runs in background and monitor devices as a daemon process. This is an experimental feature. Not supported on Windows baremetal "nvidia-smi daemon -h" for more information. replay Used to replay/extract the persistent stats generated by daemon. This is an experimental feature. "nvidia-smi replay -h" for more information. Process Monitoring: pmon Displays process stats in scrolling format. "nvidia-smi pmon -h" for more information. NVLINK: nvlink Displays device nvlink information. "nvidia-smi nvlink -h" for more information. C2C: c2c Displays device C2C information. "nvidia-smi c2c -h" for more information. CLOCKS: clocks Control and query clock information. "nvidia-smi clocks -h" for more information. ENCODER SESSIONS: encodersessions Displays device encoder sessions information. "nvidia-smi encodersessions -h" for more information. FBC SESSIONS: fbcsessions Displays device FBC sessions information. "nvidia-smi fbcsessions -h" for more information. MIG: mig Provides controls for MIG management. "nvidia-smi mig -h" for more information. COMPUTE POLICY: compute-policy Control and query compute policies. "nvidia-smi compute-policy -h" for more information. BOOST SLIDER: boost-slider Control and query boost sliders. "nvidia-smi boost-slider -h" for more information. POWER HINT: power-hint Estimates GPU power usage. "nvidia-smi power-hint -h" for more information. BASE CLOCKS: base-clocks Query GPU base clocks. "nvidia-smi base-clocks -h" for more information. GPU PERFORMANCE MONITORING: gpm Control and query GPU performance monitoring unit. "nvidia-smi gpm -h" for more information. PCI: pci Display device PCI information. "nvidia-smi pci -h" for more information.

Please see the nvidia-smi documentation for more detailed information.

[github-actions[bot]]

Stale issue