search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
7.42k stars 801 forks source link

build from source error for L40s #1412

Open littletomatodonkey opened 3 months ago

littletomatodonkey commented 3 months ago

System Info

CPU: x86_64 GPU: L40s TensorRT-LLM commit: 118b3d7e7bab720d8ea9cd95338da60f7512c93a cuda: 12.3 TRT version: 9.3.0.1

Who can help?

@byshiue

Information

Tasks

Reproduction

build scripts:

python3 scripts/build_wheel.py --trt_root /usr/local/tensorrt

Expected behavior

success

actual behavior

error: 

1 catastrophic error detected in the compilation of "/opt/tiger/TensorRT-LLM/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention192_half.cu".
Compilation terminated.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:13250: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention192_half.cu.o] Error 1
Catastrophic error: cannot open source file "/tmp/tmpxft_0000496f_00000000-11_decoderMaskedMultiheadAttention224_half.compute_89.cpp1.ii"

1 catastrophic error detected in the compilation of "/opt/tiger/TensorRT-LLM/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention224_half.cu".
Compilation terminated.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:13292: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention224_half.cu.o] Error 1
Catastrophic error: cannot open source file "/tmp/tmpxft_00004908_00000000-11_decoderMaskedMultiheadAttention144_bf16.compute_89.cpp1.ii"

1 catastrophic error detected in the compilation of "/opt/tiger/TensorRT-LLM/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention144_bf16.cu".
Compilation terminated.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:13138: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention144_bf16.cu.o] Error 1
Catastrophic error: cannot open source file "/tmp/tmpxft_00004964_00000000-11_decoderMaskedMultiheadAttention256_bf16.compute_89.cpp1.ii"

1 catastrophic error detected in the compilation of "/opt/tiger/TensorRT-LLM/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention256_bf16.cu".
Compilation terminated.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:13306: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention256_bf16.cu.o] Error 1
Catastrophic error: cannot open source file "/tmp/tmpxft_0000493c_00000000-11_decoderMaskedMultiheadAttention192_bf16.compute_89.cpp1.ii"

1 catastrophic error detected in the compilation of "/opt/tiger/TensorRT-LLM/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention192_bf16.cu".
Compilation terminated.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:13222: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention192_bf16.cu.o] Error 1
Catastrophic error: cannot open source file "/tmp/tmpxft_00004961_00000000-11_decoderMaskedMultiheadAttention224_bf16.compute_89.cpp1.ii"

1 catastrophic error detected in the compilation of "/opt/tiger/TensorRT-LLM/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention224_bf16.cu".
Compilation terminated.
gmake[3]: *** [tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/build.make:13264: tensorrt_llm/kernels/CMakeFiles/kernels_src.dir/decoderMaskedMultiheadAttention/decoderMaskedMultiheadAttention224_bf16.cu.o] Error 1
Catastrophic error: cannot open source file "/tmp/tmpxft_00004929_00000000-11_decoderMaskedMultiheadAttention160_bf16.compute_89.cpp1.ii"

additional notes

no

byshiue commented 3 months ago

Could you delete the cpp/build and run 

python3 scripts/build_wheel.py --trt_root /usr/local/tensorrt --clean

again?

littletomatodonkey commented 2 months ago

Could you delete the cpp/build and run

python3 scripts/build_wheel.py --trt_root /usr/local/tensorrt --clean

again?

Hi, @byshiue the same issue.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

byshiue commented 1 month ago

Sorry, but I cannot reproduce the issue. Could you share the scripts from building docker image, launch docker image and build/install the tensorrt_llm?

github-actions[bot] commented 3 weeks ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."