search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.18k stars 908 forks source link

cmake c++ build for Windows Visual Studio #606

Open ToddThomson opened 9 months ago

ToddThomson commented 9 months ago

Thank-you for this repo. It is a great addition to TensorRT.

I use Windows 11, Visual Studio 2022 and cmake for c++ development. I would like to be able to build the c++ folder using just these tools.

Right now, with v0.6.1, this is a bit painful. The main issues to clear up are:

1) Finding the TensorRT root directory: This is a trivial task in cmake. Right now the code is Linux only. The coarse workaround I use to get the build to work is:

# TRT dependencies
set_ifndef(TRT_LIB_DIR "C:/Program Files/NVIDIA GPU Computing Toolkit/TensorRT/v9.2.0.5/lib")
set_ifndef(TRT_INCLUDE_DIR "C:/Program Files/NVIDIA GPU Computing Toolkit/TensorRT/v9.2.0.5/include")

2) NCCL Dependencies/Setting NCCL_LIB: This is not available for Windows from reading through the readme files. The coarse workaround I use:

set_ifndef(ENABLE_MULTI_DEVICE 0)

I set the NCCL_LIB to c:/Program Files as a placeholder in the cmake project configuration file.

My ask is to be able to build the cpp folder using cmake without depending on the python build wheel script from the parent directory.

ToddThomson commented 9 months ago

Additionally...

With the above changes, I can compile the sources, but with a debug build the link stage fails due to the batch_manager static library is non-debug:

E:\GitHub\TensorRT-LLM\cpp\out\build\x64-Debug\tensorrt_llm_batch_manager_static.lib(kvCacheManager.cpp.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in assert.cpp.obj
E:\GitHub\TensorRT-LLM\cpp\out\build\x64-Debug\tensorrt_llm_batch_manager_static.lib(kvCacheManager.cpp.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MDd_DynamicDebug' in assert.cpp.obj
     Creating library tensorrt_llm\tensorrt_llm.lib and object tensorrt_llm\tensorrt_llm.exp
byshiue commented 9 months ago

Related to https://github.com/NVIDIA/TensorRT-LLM/issues/618