search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.24k stars 913 forks source link

Looking for a CUDA compiler - NOTFOUND in Dockerfile #1085

Closed faust58 closed 7 months ago

faust58 commented 7 months ago

System Info

CPU x2 Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz 128 Gb GPU x2 NVIDIA RTX A5000 24Gb OS Windows server 2022 Standart x64 branch rel

Who can help?

No response

Information

Tasks

Reproduction

PS C:\WINDOWS\system32> cd C:\TensorRT-LLM\windows\docker PS C:\TensorRT-LLM\windows\docker> docker build -t tensorrt-llm-windows-build:latest . Sending build context to Docker daemon 1.061GB Step 1/27 : FROM mcr.microsoft.com/windows/servercore:ltsc2022 ltsc2022: Pulling from windows/servercore d7662b0a97ab: Pull complete 4abd29fcbfc5: Pull complete Digest: sha256:8a75266be74ad7a904470e18057f6fd62055cf7028172307fefc67aff37dfd10 Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2022 ---> 1f57f3b65348 Step 2/27 : SHELL ["cmd", "/S", "/C"] ---> Running in f1bfdfc12506 ---> Removed intermediate container f1bfdfc12506 ---> 14e3c96c69ee Step 3/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://developer.download.nvidia.com/compute/cuda/12.2.2/local_installers/cuda_12.2.2_537.13_windows.exe -OutFile "cuda_installer.exe"; Start-Process cuda_installer.exe -Wait -ArgumentList '-s'; Remove-Item cuda_installer.exe -Force ---> Running in 3dac44bd657c ---> Removed intermediate container 3dac44bd657c ---> 87c34c8964d0 Step 4/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe -OutFile python-3.10.11.exe ; Start-Process python-3.10.11.exe -Wait -ArgumentList '/quiet InstallAllUsers=1 PrependPath=1' ; Remove-Item python-3.10.11.exe -Force ---> Running in a8c92f62803f ---> Removed intermediate container a8c92f62803f ---> 47d3697b60f9 Step 5/27 : RUN powershell -Command cp "\"C:\\Program Files\\Python310\\python.exe\" \"C:\\Program Files\\Python310\\python3.exe\"" ---> Running in 15d5ce2c6799 ---> Removed intermediate container 15d5ce2c6799 ---> ee22affddde6 Step 6/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://github.com/microsoft/Microsoft-MPI/releases/download/v10.1.1/msmpisetup.exe -OutFile "msmpisetup.exe"; Start-Process .\msmpisetup.exe -Wait ; Remove-Item msmpisetup.exe -Force ---> Running in abe0160fe671 ---> Removed intermediate container abe0160fe671 ---> 35ba1db4fbe0 Step 7/27 : RUN setx Path "%Path%;C:\Program Files\Microsoft MPI\Bin" ---> Running in a76d93d87122

SUCCESS: Specified value was saved. ---> Removed intermediate container a76d93d87122 ---> 65fae94b35a8 Step 8/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://github.com/microsoft/Microsoft-MPI/releases/download/v10.1.1/msmpisdk.msi -OutFile "msmpisdk.msi"; Start-Process msiexec.exe -Wait -ArgumentList '/I msmpisdk.msi /quiet'; Remove-Item msmpisdk.msi -Force ---> Running in d25a4d4088af ---> Removed intermediate container d25a4d4088af ---> fce47e7071fe Step 9/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://github.com/Kitware/CMake/releases/download/v3.27.7/cmake-3.27.7-windows-x86_64.msi -OutFile "cmake.msi"; Start-Process msiexec.exe -Wait -ArgumentList '/I cmake.msi /quiet'; Remove-Item cmake.msi -Force ---> Running in 6b9859feb170 ---> Removed intermediate container 6b9859feb170 ---> 46121652a723 Step 10/27 : RUN setx Path "%Path%;C:\Program Files\CMake\bin" ---> Running in 53ef849b9347

SUCCESS: Specified value was saved. ---> Removed intermediate container 53ef849b9347 ---> 926305328c35 Step 11/27 : RUN curl -SL --output vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe && (start /w vs_buildtools.exe --quiet --wait --norestart --nocache --installPath "%ProgramFiles(x86)%\Microsoft Visual Studio\2022\BuildTools" --includeRecommended --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Workload.VCTools --remove Microsoft.VisualStudio.Component.Windows10SDK.10240 --remove Microsoft.VisualStudio.Component.Windows10SDK.10586 --remove Microsoft.VisualStudio.Component.Windows10SDK.14393 --remove Microsoft.VisualStudio.Component.Windows81SDK || IF "%ERRORLEVEL%"=="3010" EXIT 0) && del /q vs_buildtools.exe ---> Running in 9c56b0f744be % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 3902k 100 3902k 0 0 2923k 0 0:00:01 0:00:01 --:--:-- 10.4M ---> Removed intermediate container 9c56b0f744be ---> 1d37cb4bc870 Step 12/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://ftp.nluug.nl/pub/vim/pc/gvim90.exe -OutFile "install_vim.exe"; Start-Process install_vim.exe -Wait -ArgumentList '/S'; Remove-Item install_vim.exe -Force ---> Running in 23a9a9227647 ---> Removed intermediate container 23a9a9227647 ---> cbc16515824a Step 13/27 : RUN setx Path "%Path%;C:\Program Files (x86)\Vim\vim90" ---> Running in 90d0bbbaea67

SUCCESS: Specified value was saved. ---> Removed intermediate container 90d0bbbaea67 ---> b8f7f640079d Step 14/27 : ENV chocolateyVersion=1.4.0 ---> Running in 7c01ad351daf ---> Removed intermediate container 7c01ad351daf ---> f9b1a94b08ed Step 15/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; powershell.exe -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "[System.Net.ServicePointManager]::SecurityProtocol = 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin" ---> Running in 8bddbc4be750 Forcing web requests to allow TLS v1.2 (Required for requests to Chocolatey.org) Downloading specific version of Chocolatey: 1.4.0 Getting Chocolatey from https://community.chocolatey.org/api/v2/package/chocolatey/1.4.0. Downloading https://community.chocolatey.org/api/v2/package/chocolatey/1.4.0 to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip Not using proxy. Extracting C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip to C:\Users\ContainerAdministrator\AppData\Local\Temp\chocolatey\chocoInstall Installing Chocolatey on the local machine Creating ChocolateyInstall as an environment variable (targeting 'Machine') Setting ChocolateyInstall to 'C:\ProgramData\chocolatey' WARNING: It's very likely you will need to close and reopen your shell before you can use choco. Restricting write permissions to Administrators We are setting up the Chocolatey package repository. The packages themselves go to 'C:\ProgramData\chocolatey\lib' (i.e. C:\ProgramData\chocolatey\lib\yourPackageName). A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin' and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.

Creating Chocolatey folders if they do not already exist.

WARNING: You can safely ignore errors related to missing log files when upgrading from a version of Chocolatey less than 0.9.9. 'Batch file could not be found' is also safe to ignore. 'The system cannot find the file specified' - also safe. chocolatey.nupkg file not installed in lib. Attempting to locate it from bootstrapper. PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding... WARNING: Not setting tab completion: Profile file does not exist at 'C:\Users\ContainerAdministrator\Documents\WindowsPowerShell\Microsoft.PowerShe ll_profile.ps1'. Chocolatey (choco.exe) is now ready. You can call choco from anywhere, command line or powershell by typing choco. Run choco /? for a list of functions. You may need to shut down and restart powershell and/or consoles first prior to using choco. Ensuring Chocolatey commands are on the path Ensuring chocolatey.nupkg is in the lib folder ---> Removed intermediate container 8bddbc4be750 ---> 561687d93b43 Step 16/27 : RUN powershell -Command choco install git -y ---> Running in 5da012c3b7d5 Chocolatey v1.4.0 Installing the following packages: git By installing, you accept licenses for the packages. Progress: Downloading git.install 2.43.0... 100% Progress: Downloading chocolatey-core.extension 1.4.0... 100% Progress: Downloading chocolatey-compatibility.extension 1.0.0... 100% Progress: Downloading git 2.43.0... 100%

chocolatey-compatibility.extension v1.0.0 [Approved] chocolatey-compatibility.extension package files install completed. Performing other installation steps. Installed/updated chocolatey-compatibility extensions. The install of chocolatey-compatibility.extension was successful. Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-compatibility'

chocolatey-core.extension v1.4.0 [Approved] chocolatey-core.extension package files install completed. Performing other installation steps. Installed/updated chocolatey-core extensions. The install of chocolatey-core.extension was successful. Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

git.install v2.43.0 [Approved] git.install package files install completed. Performing other installation steps. Using Git LFS Installing 64-bit git.install... git.install has been installed. git.install installed to 'C:\Program Files\Git' git.install can be automatically uninstalled. Environment Vars (like PATH) have changed. Close/reopen your shell to see the changes (or in powershell/cmd.exe just type refreshenv). The install of git.install was successful. Software installed to 'C:\Program Files\Git\'

git v2.43.0 [Approved] git package files install completed. Performing other installation steps. The install of git was successful. Software installed to 'C:\ProgramData\chocolatey\lib\git'

Chocolatey installed 4/4 packages. See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Did you know the proceeds of Pro (and some proceeds from other licensed editions) go into bettering the community infrastructure? Your support ensures an active community, keeps Chocolatey tip-top, plus it nets you some awesome features! https://chocolatey.org/compare ---> Removed intermediate container 5da012c3b7d5 ---> df994e935fe3 Step 17/27 : COPY ["NvToolsExt", "C:\\Program Files\\NVIDIA Corporation\\NvToolsExt"] ---> e666a84780eb Step 18/27 : RUN setx Path "%Path%;C:\Program Files\NVIDIA Corporation\NvToolsExt;" ---> Running in 4c9fca0a15e8

SUCCESS: Specified value was saved. ---> Removed intermediate container 4c9fca0a15e8 ---> ceec3996505f Step 19/27 : WORKDIR "C:\\workspace" ---> Running in 8d6b03b1cf5c ---> Removed intermediate container 8d6b03b1cf5c ---> f730ae4afcdb Step 20/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; Invoke-WebRequest -Uri https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/9.2.0/tensorrt-9.2.0.5.windows10.x86_64.cuda-12.2.llm.beta.zip -OutFile TensorRT-9.2.0.5.zip; Expand-Archive .\TensorRT-9.2.0.5.zip -DestinationPath .; Remove-Item TensorRT-9.2.0.5.zip -Force ---> Running in c98210e75a05 ---> Removed intermediate container c98210e75a05 ---> 5ff23a0f8b4c Step 21/27 : RUN setx Path "%Path%;C:\workspace\TensorRT-9.2.0.5\lib" ---> Running in 79d3c8f8ea46

SUCCESS: Specified value was saved. ---> Removed intermediate container 79d3c8f8ea46 ---> 482e13b81dae Step 22/27 : RUN powershell -Command $ErrorActionPreference = 'Stop'; pip install TensorRT-9.2.0.5\python\tensorrt-9.2.0.post12.dev5-cp310-none-win_amd64.whl ---> Running in 398fde20faa9 Processing c:\workspace\tensorrt-9.2.0.5\python\tensorrt-9.2.0.post12.dev5-cp310-none-win_amd64.whl Installing collected packages: tensorrt Successfully installed tensorrt-9.2.0.post12.dev5

[notice] A new release of pip is available: 23.0.1 -> 24.0 [notice] To update, run: python.exe -m pip install --upgrade pip ---> Removed intermediate container 398fde20faa9 ---> e9bc86632a42 Step 23/27 : COPY ["cuDNN", "cuDNN"] ---> bb998364bd25 Step 24/27 : RUN setx Path "%Path%;C:\workspace\cuDNN\lib;C:\workspace\cuDNN\bin;" ---> Running in eac4cfd021f1

SUCCESS: Specified value was saved. ---> Removed intermediate container eac4cfd021f1 ---> 0134a7bc7880 Step 25/27 : COPY ["nvcc.exe", "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.2\\bin"] ---> 07fb610ef826 Step 26/27 : RUN setx Path "%Path%;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin;" ---> Running in 9077f413ccf0

SUCCESS: Specified value was saved. ---> Removed intermediate container 9077f413ccf0 ---> ba26ef5b8e41 Step 27/27 : ENTRYPOINT ["C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\Common7\Tools\VsDevCmd.bat", "-arch=amd64", "&&", "powershell.exe", "-NoLogo", "-ExecutionPolicy", "Bypass"] ---> Running in 7a00f81226e1 ---> Removed intermediate container 7a00f81226e1 ---> ece98a218f19 Successfully built ece98a218f19 Successfully tagged tensorrt-llm-windows-build:latest

What's Next? View a summary of image vulnerabilities and recommendations → docker scout quickview PS C:\TensorRT-LLM\windows\docker> docker run -it -m 12g -v .\trt-llm-build:C:\workspace\trt-llm-build tensorrt-llm-windows-build:latest

Visual Studio 2022 Developer Command Prompt v17.9.0 Copyright (c) 2022 Microsoft Corporation

PS C:\workspace> git clone --branch rel https://github.com/NVIDIA/TensorRT-LLM.git Cloning into 'TensorRT-LLM'... remote: Enumerating objects: 9096, done. remote: Counting objects: 100% (2850/2850), done. remote: Compressing objects: 100% (926/926), done. remote: Total 9096 (delta 2018), reused 2508 (delta 1912), pack-reused 6246Receiving objects: 100% (9096/9096), 123.32 MReceiving objec

Resolving deltas: 100% (6289/6289), done. Updating files: 100% (1324/1324), done. Filtering content: 100% (5/5), 18.61 MiB | 4.70 MiB/s, done. PS C:\workspace> cd TensorRT-LLM PS C:\workspace\TensorRT-LLM> git submodule update --init --recursive Submodule '3rdparty/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path '3rdparty/NVTX' Submodule '3rdparty/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path '3rdparty/cutlass' Submodule '3rdparty/cxxopts' (https://github.com/jarro2783/cxxopts) registered for path '3rdparty/cxxopts' Submodule '3rdparty/json' (https://github.com/nlohmann/json.git) registered for path '3rdparty/json' Cloning into 'C:/workspace/TensorRT-LLM/3rdparty/NVTX'... Cloning into 'C:/workspace/TensorRT-LLM/3rdparty/cutlass'... Cloning into 'C:/workspace/TensorRT-LLM/3rdparty/cxxopts'... Cloning into 'C:/workspace/TensorRT-LLM/3rdparty/json'... Submodule path '3rdparty/NVTX': checked out 'a1ceb0677f67371ed29a2b1c022794f077db5fe7' Submodule path '3rdparty/cutlass': checked out '39c6a83f231d6db2bc6b9c251e7add77d68cbfb4' Submodule path '3rdparty/cxxopts': checked out 'eb787304d67ec22f7c3a184ee8b4c481d04357fd' Submodule path '3rdparty/json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' PS C:\workspace\TensorRT-LLM> python .\scripts\build_wheel.py -a "89-real" --trt_root C:\workspace\TensorRT-9.2.0.5\ Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com, https://download.pytorch.org/whl/cu121, https://pypi.nvidia.com, https://download.pytorch.org/whl/cu121 Collecting accelerate==0.20.3 Downloading accelerate-0.20.3-py3-none-any.whl (227 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 2.8 MB/s eta 0:00:00 Collecting build Downloading build-1.0.3-py3-none-any.whl (18 kB) Collecting colored Downloading colored-2.2.4-py3-none-any.whl (16 kB) Collecting cuda-python==12.2.0 Downloading cuda_python-12.2.0-cp310-cp310-win_amd64.whl (8.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.3/8.3 MB 1.3 MB/s eta 0:00:00 Collecting diffusers==0.15.0 Downloading diffusers-0.15.0-py3-none-any.whl (851 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 851.8/851.8 kB 1.9 MB/s eta 0:00:00 Collecting mpi4py Downloading mpi4py-3.1.5-cp310-cp310-win_amd64.whl (472 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 472.1/472.1 kB 1.5 MB/s eta 0:00:00 Collecting numpy Downloading numpy-1.26.4-cp310-cp310-win_amd64.whl (15.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.8/15.8 MB 1.1 MB/s eta 0:00:00 Collecting onnx>=1.12.0 Downloading onnx-1.15.0-cp310-cp310-win_amd64.whl (14.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.3/14.3 MB 1.4 MB/s eta 0:00:00 Collecting polygraphy Downloading https://pypi.nvidia.com/polygraphy/polygraphy-0.49.0-py2.py3-none-any.whl (327 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 327.9/327.9 kB 3.4 MB/s eta 0:00:00 Collecting pywin32 Downloading pywin32-306-cp310-cp310-win_amd64.whl (9.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.2/9.2 MB 1.3 MB/s eta 0:00:00 Collecting sentencepiece>=0.1.99 Downloading sentencepiece-0.1.99-cp310-cp310-win_amd64.whl (977 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 977.5/977.5 kB 1.4 MB/s eta 0:00:00 Requirement already satisfied: tensorrt==9.2.0.post12.dev5 in c:\program files\python310\lib\site-packages (from -r requirements-windows.txt (line 14)) (9.2.0.post12.dev5) Collecting tokenizers==0.13.3 Downloading tokenizers-0.13.3-cp310-cp310-win_amd64.whl (3.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 2.6 MB/s eta 0:00:00 Collecting torch==2.1.0+cu121 Downloading https://download.pytorch.org/whl/cu121/torch-2.1.0%2Bcu121-cp310-cp310-win_amd64.whl (2473.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 GB 721.5 kB/s eta 0:00:00 Collecting torchdata==0.7.0 Downloading https://download.pytorch.org/whl/torchdata-0.7.0-cp310-cp310-win_amd64.whl (1.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 8.5 MB/s eta 0:00:00 Collecting torchtext==0.16.0+cpu Downloading https://download.pytorch.org/whl/torchtext-0.16.0%2Bcpu-cp310-cp310-win_amd64.whl (1.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 12.4 MB/s eta 0:00:00 Collecting torchvision==0.16.0+cu121 Downloading https://download.pytorch.org/whl/cu121/torchvision-0.16.0%2Bcu121-cp310-cp310-win_amd64.whl (5.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.8/5.8 MB 12.3 MB/s eta 0:00:00 Collecting transformers==4.33.1 Downloading transformers-4.33.1-py3-none-any.whl (7.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 1.1 MB/s eta 0:00:00 Collecting wheel Downloading wheel-0.42.0-py3-none-any.whl (65 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.4/65.4 kB 1.7 MB/s eta 0:00:00 Collecting einops Downloading einops-0.7.0-py3-none-any.whl (44 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 2.1 MB/s eta 0:00:00 Collecting graphviz Downloading graphviz-0.20.1-py3-none-any.whl (47 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.0/47.0 kB 334.5 kB/s eta 0:00:00 Collecting mypy Downloading mypy-1.8.0-cp310-cp310-win_amd64.whl (9.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.2/9.2 MB 1.5 MB/s eta 0:00:00 Collecting parameterized Downloading parameterized-0.9.0-py2.py3-none-any.whl (20 kB) Collecting pre-commit Downloading pre_commit-3.6.1-py2.py3-none-any.whl (204 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 204.2/204.2 kB 2.5 MB/s eta 0:00:00 Collecting pybind11-stubgen Downloading pybind11_stubgen-2.4.2-py3-none-any.whl (29 kB) Collecting pynvml>=11.5.0 Downloading pynvml-11.5.0-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 1.4 MB/s eta 0:00:00 Collecting pytest-cov Downloading pytest_cov-4.1.0-py3-none-any.whl (21 kB) Collecting pytest-forked Downloading pytest_forked-1.6.0-py3-none-any.whl (4.9 kB) Collecting pytest-xdist Downloading pytest_xdist-3.5.0-py3-none-any.whl (42 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.0/42.0 kB 1.0 MB/s eta 0:00:00 Collecting typing-extensions==4.8.0 Downloading https://download.pytorch.org/whl/typing_extensions-4.8.0-py3-none-any.whl (31 kB) Collecting packaging>=20.0 Downloading packaging-23.2-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/53.0 kB 1.4 MB/s eta 0:00:00 Collecting pyyaml Downloading PyYAML-6.0.1-cp310-cp310-win_amd64.whl (145 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 145.3/145.3 kB 1.2 MB/s eta 0:00:00 Collecting psutil Downloading psutil-5.9.8-cp37-abi3-win_amd64.whl (255 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 255.1/255.1 kB 1.4 MB/s eta 0:00:00 Collecting cython Downloading Cython-3.0.8-cp310-cp310-win_amd64.whl (2.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 1.6 MB/s eta 0:00:00 Collecting regex!=2019.12.17 Downloading regex-2023.12.25-cp310-cp310-win_amd64.whl (269 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 269.5/269.5 kB 1.2 MB/s eta 0:00:00 Collecting Pillow Downloading https://download.pytorch.org/whl/pillow-10.2.0-cp310-cp310-win_amd64.whl (2.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 3.6 MB/s eta 0:00:00 Collecting filelock Downloading filelock-3.13.1-py3-none-any.whl (11 kB) Collecting huggingface-hub>=0.13.2 Downloading huggingface_hub-0.20.3-py3-none-any.whl (330 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 330.1/330.1 kB 1.0 MB/s eta 0:00:00 Collecting requests Downloading requests-2.31.0-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 830.7 kB/s eta 0:00:00 Collecting importlib-metadata Downloading importlib_metadata-7.0.1-py3-none-any.whl (23 kB) Collecting jinja2 Downloading Jinja2-3.1.3-py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.2/133.2 kB 878.5 kB/s eta 0:00:00 Collecting sympy Downloading https://download.pytorch.org/whl/sympy-1.12-py3-none-any.whl (5.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 1.2 MB/s eta 0:00:00 Collecting fsspec Downloading fsspec-2024.2.0-py3-none-any.whl (170 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 170.9/170.9 kB 854.8 kB/s eta 0:00:00 Collecting networkx Downloading https://download.pytorch.org/whl/networkx-3.2.1-py3-none-any.whl (1.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 895.3 kB/s eta 0:00:00 Collecting urllib3>=1.25 Downloading urllib3-2.2.0-py3-none-any.whl (120 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.9/120.9 kB 885.1 kB/s eta 0:00:00 Collecting tqdm Downloading tqdm-4.66.2-py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 1.1 MB/s eta 0:00:00 Collecting safetensors>=0.3.1 Downloading safetensors-0.4.2-cp310-none-win_amd64.whl (269 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 269.5/269.5 kB 1.1 MB/s eta 0:00:00 Collecting colorama Downloading https://download.pytorch.org/whl/colorama-0.4.6-py2.py3-none-any.whl (25 kB) Collecting tomli>=1.1.0 Downloading tomli-2.0.1-py3-none-any.whl (12 kB) Collecting pyproject_hooks Downloading pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB) Collecting protobuf>=3.20.2 Downloading protobuf-4.25.2-cp310-abi3-win_amd64.whl (413 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 413.4/413.4 kB 1.4 MB/s eta 0:00:00 Collecting mypy-extensions>=1.0.0 Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB) Collecting cfgv>=2.0.0 Downloading cfgv-3.4.0-py2.py3-none-any.whl (7.2 kB) Collecting virtualenv>=20.10.0 Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 1.9 MB/s eta 0:00:00 Collecting nodeenv>=0.11.1 Downloading nodeenv-1.8.0-py2.py3-none-any.whl (22 kB) Collecting identify>=1.0.0 Downloading identify-2.5.34-py2.py3-none-any.whl (98 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.9/98.9 kB 2.8 MB/s eta 0:00:00 Collecting pytest>=4.6 Downloading pytest-8.0.0-py3-none-any.whl (334 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 334.0/334.0 kB 2.9 MB/s eta 0:00:00 Collecting coverage[toml]>=5.2.1 Downloading coverage-7.4.1-cp310-cp310-win_amd64.whl (209 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.7/209.7 kB 2.1 MB/s eta 0:00:00 Collecting py Downloading py-1.11.0-py2.py3-none-any.whl (98 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.7/98.7 kB 1.9 MB/s eta 0:00:00 Collecting execnet>=1.1 Downloading execnet-2.0.2-py3-none-any.whl (37 kB) Requirement already satisfied: setuptools in c:\program files\python310\lib\site-packages (from nodeenv>=0.11.1->pre-commit->-r requirements-dev-windows.txt (line 7)) (65.5.0) Collecting exceptiongroup>=1.0.0rc8 Downloading exceptiongroup-1.2.0-py3-none-any.whl (16 kB) Collecting pluggy<2.0,>=1.3.0 Downloading pluggy-1.4.0-py3-none-any.whl (20 kB) Collecting iniconfig Downloading iniconfig-2.0.0-py3-none-any.whl (5.9 kB) Collecting platformdirs<5,>=3.9.1 Downloading platformdirs-4.2.0-py3-none-any.whl (17 kB) Collecting distlib<1,>=0.3.7 Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 kB 2.1 MB/s eta 0:00:00 Collecting zipp>=0.5 Downloading zipp-3.17.0-py3-none-any.whl (7.4 kB) Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.5-cp310-cp310-win_amd64.whl (17 kB) Collecting charset-normalizer<4,>=2 Downloading charset_normalizer-3.3.2-cp310-cp310-win_amd64.whl (100 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.3/100.3 kB 1.9 MB/s eta 0:00:00 Collecting idna<4,>=2.5 Downloading idna-3.6-py3-none-any.whl (61 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 3.2 MB/s eta 0:00:00 Collecting certifi>=2017.4.17 Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 2.5 MB/s eta 0:00:00 Collecting mpmath>=0.19 Downloading https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl (536 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 4.8 MB/s eta 0:00:00 Installing collected packages: tokenizers, sentencepiece, pywin32, mpmath, distlib, zipp, wheel, urllib3, typing-extensions, tomli, sympy, safetensors, regex, pyyaml, pynvml, pybind11-stubgen, py, psutil, protobuf, polygraphy, pluggy, platformdirs, Pillow, parameterized, packaging, numpy, nodeenv, networkx, mypy-extensions, mpi4py, MarkupSafe, iniconfig, idna, identify, graphviz, fsspec, filelock, execnet, exceptiongroup, einops, cython, coverage, colored, colorama, charset-normalizer, cfgv, certifi, virtualenv, tqdm, requests, pytest, pyproject_hooks, onnx, mypy, jinja2, importlib-metadata, cuda-python, torch, pytest-xdist, pytest-forked, pytest-cov, pre-commit, huggingface-hub, build, transformers, torchvision, torchdata, diffusers, accelerate, torchtext Successfully installed MarkupSafe-2.1.5 Pillow-10.2.0 accelerate-0.20.3 build-1.0.3 certifi-2024.2.2 cfgv-3.4.0 charset-normalizer-3.3.2 colorama-0.4.6 colored-2.2.4 coverage-7.4.1 cuda-python-12.2.0 cython-3.0.8 diffusers-0.15.0 distlib-0.3.8 einops-0.7.0 exceptiongroup-1.2.0 execnet-2.0.2 filelock-3.13.1 fsspec-2024.2.0 graphviz-0.20.1 huggingface-hub-0.20.3 identify-2.5.34 idna-3.6 importlib-metadata-7.0.1 iniconfig-2.0.0 jinja2-3.1.3 mpi4py-3.1.5 mpmath-1.3.0 mypy-1.8.0 mypy-extensions-1.0.0 networkx-3.2.1 nodeenv-1.8.0 numpy-1.26.4 onnx-1.15.0 packaging-23.2 parameterized-0.9.0 platformdirs-4.2.0 pluggy-1.4.0 polygraphy-0.49.0 pre-commit-3.6.1 protobuf-4.25.2 psutil-5.9.8 py-1.11.0 pybind11-stubgen-2.4.2 pynvml-11.5.0 pyproject_hooks-1.0.0 pytest-8.0.0 pytest-cov-4.1.0 pytest-forked-1.6.0 pytest-xdist-3.5.0 pywin32-306 pyyaml-6.0.1 regex-2023.12.25 requests-2.31.0 safetensors-0.4.2 sentencepiece-0.1.99 sympy-1.12 tokenizers-0.13.3 tomli-2.0.1 torch-2.1.0+cu121 torchdata-0.7.0 torchtext-0.16.0+cpu torchvision-0.16.0+cu121 tqdm-4.66.2 transformers-4.33.1 typing-extensions-4.8.0 urllib3-2.2.0 virtualenv-20.25.0 wheel-0.42.0 zipp-3.17.0

[notice] A new release of pip is available: 23.0.1 -> 24.0 [notice] To update, run: python.exe -m pip install --upgrade pip -- The CXX compiler identification is MSVC 19.39.33519.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- NVTX is disabled -- Importing batch manager -- Building PyTorch -- Building Google tests -- Building benchmarks -- Looking for a CUDA compiler -- Looking for a CUDA compiler - NOTFOUND CMake Error at CMakeLists.txt:118 (message): No CUDA compiler found

-- Configuring incomplete, errors occurred! Traceback (most recent call last): File "C:\workspace\TensorRT-LLM\scripts\build_wheel.py", line 306, in main(**vars(args)) File "C:\workspace\TensorRT-LLM\scripts\build_wheel.py", line 160, in main build_run( File "C:\Program Files\Python310\lib\subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command 'cmake -DCMAKE_BUILD_TYPE="Release" -DBUILD_PYT="ON" -DBUILD_PYBIND="OFF" "-DCMAKE_CUDA_ARCHITECTURES=89-real" "-DENABLE_MULTI_DEVICE=0" -DTRT_LIB_DIR=C:/workspace/TensorRT-9.2.0.5//lib -DTRT_INCLUDE_DIR=C:/workspace/TensorRT-9.2.0.5//include -GNinja -S "C:\workspace\TensorRT-LLM\cpp"' returned non-zero exit status 1. PS C:\workspace\TensorRT-LLM>

Expected behavior

python .\scripts\build_wheel.py -a "89-real" --trt_root

actual behavior

-- Looking for a CUDA compiler -- Looking for a CUDA compiler - NOTFOUND CMake Error at CMakeLists.txt:118 (message): No CUDA compiler found

additional notes

Please help me solve a problem. I tried the "Bare-Metal Build Instructions", but I'm also getting errors.

htdung167 commented 4 months ago

I had the same problem, how did you solve it?

faust58 commented 4 months ago

@htdung167 I did not fulfill this requirement during the installation: Download and install Visual Studio 2022. When prompted to select more Workloads, check Desktop development with C++.

Also, check the compatibility of the GPU driver version with CUDA; I had v550 installed, but for CUDA 12.2, v535 is needed.