Open BasicCoder opened 9 months ago
Got it here as well, also TP4 70b, I'm able to reproduce it over large parallel batch requests. https://github.com/NVIDIA/TensorRT-LLM/issues/427
@basiccoder are you using a quantization? I'm using fp8 but seems like the issue doesn't have to do with the quants.
@byshiue think you're right it may be easier to debug on smaller but perhaps the issue is the tensor parallelism but if the issue is with the tensor parallelism it might not be possible to get around it. Wonder if it's just with Llama 70b too or some working of MQA? Still can send it over to you.
@BasicCoder @0xymoro Does this issue happen on python runtime?
@BasicCoder @0xymoro Does this issue happen on python runtime?
Sorry, I did not try the python runtime. This error appeared after the service had been running stably for a while. After I turned off all possible optimization options and kept only these options:
python build.py --world_size 4 \
--tp_size 4 \
--model_dir /70B_models \
--dtype float16 \
--max_batch_size 4 \
--max_input_len 4096 \
--max_output_len 4096 \
--max_beam_width 1 \
--use_gpt_attention_plugin float16 \
--use_gemm_plugin float16 \
--use_rmsnorm_plugin float16 \
--use_inflight_batching \
--paged_kv_cache \
--remove_input_padding \
--output_dir /70B_models/trt_engines/fp16/4-gpu/
a new ERROR appeared:
terminate called after throwing an instance of 'tensorrt_llm::common::TllmException'
what(): [TensorRT-LLM][ERROR] CUDA runtime error in cublasLtMatmul(getCublasLtHandle(), mOperationDesc, alpha, A, mADesc, B, mBDesc, beta, C, mCDesc, C, mCDesc, (hasAlgo ? (&algo) : NULL), mCublasWorkspace, workspaceSize, mStream): CUBLAS_STATUS_EXECUTION_FAILED (/app/tensorrt_llm/cpp/tensorrt_llm/common/cublasM
MWrapper.cpp:140)
1 0x7f4cf6b89a6e /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0x8ea6e) [0x7f4cf6b89a6e]
2 0x7f4cf6bce7c5 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd37c5) [0x7f4cf6bce7c5]
3 0x7f4cf6bceb7b /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd3b7b) [0x7f4cf6bceb7b]
4 0x7f4cf6ba7e81 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xace81) [0x7f4cf6ba7e81]
5 0x7f4cf6ba8767 tensorrt_llm::plugins::GemmPlugin::enqueue(nvinfer1::PluginTensorDesc const*, nvinfer1::PluginTensorDesc const*, void const* const*, void* const*, void*, CUstream_st*) + 2636 0x7f4d34afbfc9 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10aefc9) [0x7f4d34afbfc9]
7 0x7f4d34abee04 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1071e04) [0x7f4d34abee04]
8 0x7f4d34ac09a0 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10739a0) [0x7f4d34ac09a0]
9 0x7f4d4341ec37 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5cc37) [0x7f4d4341ec37]
10 0x7f4d434203ed /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5e3ed) [0x7f4d434203ed]
11 0x7f4d4342312d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6112d) [0x7f4d4342312d]
12 0x7f4d43413aa1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x51aa1) [0x7f4d43413aa1]
13 0x7f4d43415902 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x53902) [0x7f4d43415902]14 0x7f4ee0e72253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f4ee0e72253]
15 0x7f4ee0c02b43 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ee0c02b43]
16 0x7f4ee0c94a00 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ee0c94a00]
terminate called after throwing an instance of 'tensorrt_llm::common::TllmException'
what(): [TensorRT-LLM][ERROR] CUDA runtime error in cublasLtMatmul(getCublasLtHandle(), mOperationDesc, alpha, A, mADesc, B, mBDesc, beta, C, mCDesc, C, mCDesc, (hasAlgo ? (&algo) : NULL), mCublasWorkspace, workspaceSize, mStream): CUBLAS_STATUS_EXECUTION_FAILED (/app/tensorrt_llm/cpp/tensorrt_llm/common/cublasM
MWrapper.cpp:140)
1 0x7f9a26b89a6e /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0x8ea6e) [0x7f9a26b89a6e]
2 0x7f9a26bce7c5 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd37c5) [0x7f9a26bce7c5]
3 0x7f9a26bceb7b /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd3b7b) [0x7f9a26bceb7b]
4 0x7f9a26ba7e81 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xace81) [0x7f9a26ba7e81]
5 0x7f9a26ba8767 tensorrt_llm::plugins::GemmPlugin::enqueue(nvinfer1::PluginTensorDesc const*, nvinfer1::PluginTensorDesc const*, void const* const*, void* const*, void*, CUstream_st*) + 263
6 0x7f9a64afbfc9 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10aefc9) [0x7f9a64afbfc9]
7 0x7f9a64abee04 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1071e04) [0x7f9a64abee04]
8 0x7f9a64ac09a0 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10739a0) [0x7f9a64ac09a0]
9 0x7f9a7341ec37 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5cc37) [0x7f9a7341ec37]
10 0x7f9a734203ed /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5e3ed) [0x7f9a734203ed]
11 0x7f9a7342312d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6112d) [0x7f9a7342312d]
12 0x7f9a73413aa1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x51aa1) [0x7f9a73413aa1]
13 0x7f9a73415902 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x53902) [0x7f9a73415902]
14 0x7f9c0f872253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f9c0f872253]
15 0x7f9c0f602b43 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f9c0f602b43]
16 0x7f9c0f694a00 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9c0f694a00]
Signal (6) received.
[archimedes-open-api-70b-1325343-5875b79c44-m6l66:06554] *** Process received signal ***
[archimedes-open-api-70b-1325343-5875b79c44-m6l66:06554] Signal: Aborted (6)
[archimedes-open-api-70b-1325343-5875b79c44-m6l66:06554] Signal code: (-6)
This new error also appeared after the service had been running stably for some time.
After nearly two weeks of testing, the most stable engine construction method is:
python build.py --world_size 4 \
--tp_size 4 \
--model_dir /70B_models \
--dtype float16 \
--max_batch_size 4 \
--max_input_len 4096 \
--max_output_len 4096 \
--max_beam_width 1 \
--use_gpt_attention_plugin float16 \
--use_gemm_plugin float16 \
--use_inflight_batching \
--paged_kv_cache \
--remove_input_padding \
--output_dir /70B_models/trt_engines/fp16/4-gpu/
@BasicCoder @0xymoro Does this issue happen on python runtime?
Sorry, I did not try the python runtime. This error appeared after the service had been running stably for a while. After I turned off all possible optimization options and kept only these options:
python build.py --world_size 4 \ --tp_size 4 \ --model_dir /70B_models \ --dtype float16 \ --max_batch_size 4 \ --max_input_len 4096 \ --max_output_len 4096 \ --max_beam_width 1 \ --use_gpt_attention_plugin float16 \ --use_gemm_plugin float16 \ --use_rmsnorm_plugin float16 \ --use_inflight_batching \ --paged_kv_cache \ --remove_input_padding \ --output_dir /70B_models/trt_engines/fp16/4-gpu/
a new ERROR appeared:
terminate called after throwing an instance of 'tensorrt_llm::common::TllmException' what(): [TensorRT-LLM][ERROR] CUDA runtime error in cublasLtMatmul(getCublasLtHandle(), mOperationDesc, alpha, A, mADesc, B, mBDesc, beta, C, mCDesc, C, mCDesc, (hasAlgo ? (&algo) : NULL), mCublasWorkspace, workspaceSize, mStream): CUBLAS_STATUS_EXECUTION_FAILED (/app/tensorrt_llm/cpp/tensorrt_llm/common/cublasM MWrapper.cpp:140) 1 0x7f4cf6b89a6e /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0x8ea6e) [0x7f4cf6b89a6e] 2 0x7f4cf6bce7c5 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd37c5) [0x7f4cf6bce7c5] 3 0x7f4cf6bceb7b /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd3b7b) [0x7f4cf6bceb7b] 4 0x7f4cf6ba7e81 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xace81) [0x7f4cf6ba7e81] 5 0x7f4cf6ba8767 tensorrt_llm::plugins::GemmPlugin::enqueue(nvinfer1::PluginTensorDesc const*, nvinfer1::PluginTensorDesc const*, void const* const*, void* const*, void*, CUstream_st*) + 2636 0x7f4d34afbfc9 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10aefc9) [0x7f4d34afbfc9] 7 0x7f4d34abee04 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1071e04) [0x7f4d34abee04] 8 0x7f4d34ac09a0 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10739a0) [0x7f4d34ac09a0] 9 0x7f4d4341ec37 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5cc37) [0x7f4d4341ec37] 10 0x7f4d434203ed /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5e3ed) [0x7f4d434203ed] 11 0x7f4d4342312d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6112d) [0x7f4d4342312d] 12 0x7f4d43413aa1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x51aa1) [0x7f4d43413aa1] 13 0x7f4d43415902 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x53902) [0x7f4d43415902]14 0x7f4ee0e72253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f4ee0e72253] 15 0x7f4ee0c02b43 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ee0c02b43] 16 0x7f4ee0c94a00 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ee0c94a00] terminate called after throwing an instance of 'tensorrt_llm::common::TllmException' what(): [TensorRT-LLM][ERROR] CUDA runtime error in cublasLtMatmul(getCublasLtHandle(), mOperationDesc, alpha, A, mADesc, B, mBDesc, beta, C, mCDesc, C, mCDesc, (hasAlgo ? (&algo) : NULL), mCublasWorkspace, workspaceSize, mStream): CUBLAS_STATUS_EXECUTION_FAILED (/app/tensorrt_llm/cpp/tensorrt_llm/common/cublasM MWrapper.cpp:140) 1 0x7f9a26b89a6e /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0x8ea6e) [0x7f9a26b89a6e] 2 0x7f9a26bce7c5 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd37c5) [0x7f9a26bce7c5] 3 0x7f9a26bceb7b /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xd3b7b) [0x7f9a26bceb7b] 4 0x7f9a26ba7e81 /app/inflight_batcher_llm/../tensorrt_llm/cpp/build/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so.9(+0xace81) [0x7f9a26ba7e81] 5 0x7f9a26ba8767 tensorrt_llm::plugins::GemmPlugin::enqueue(nvinfer1::PluginTensorDesc const*, nvinfer1::PluginTensorDesc const*, void const* const*, void* const*, void*, CUstream_st*) + 263 6 0x7f9a64afbfc9 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10aefc9) [0x7f9a64afbfc9] 7 0x7f9a64abee04 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1071e04) [0x7f9a64abee04] 8 0x7f9a64ac09a0 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10739a0) [0x7f9a64ac09a0] 9 0x7f9a7341ec37 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5cc37) [0x7f9a7341ec37] 10 0x7f9a734203ed /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x5e3ed) [0x7f9a734203ed] 11 0x7f9a7342312d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6112d) [0x7f9a7342312d] 12 0x7f9a73413aa1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x51aa1) [0x7f9a73413aa1] 13 0x7f9a73415902 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x53902) [0x7f9a73415902] 14 0x7f9c0f872253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f9c0f872253] 15 0x7f9c0f602b43 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f9c0f602b43] 16 0x7f9c0f694a00 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f9c0f694a00] Signal (6) received. [archimedes-open-api-70b-1325343-5875b79c44-m6l66:06554] *** Process received signal *** [archimedes-open-api-70b-1325343-5875b79c44-m6l66:06554] Signal: Aborted (6) [archimedes-open-api-70b-1325343-5875b79c44-m6l66:06554] Signal code: (-6)
This new error also appeared after the service had been running stably for some time.
This error may be solved by setting kv_cache_free_gpu_mem_fraction=0.80
to reserve more buffers(default=0.85).
Got some free time and went back and did some older tests to see if kv cache free gpu mem did it. It did not. I manually set it to 0.65 and below, and it still errored out when stress tested with long context with many in parallel at at time.
The only difference in my engine that failed, and @BasicCoder's that they said was stable is that I have --enable_context_fmha. It might be a pretty good bet to isolate the issue on this, as to what the flash attention is doing wrong here in you guys' investigations.
My build args that resulted in fails:
--dtype float16 --use_gpt_attention_plugin float16 --use_gemm_plugin float16 --remove_input_padding --use_inflight_batching --paged_kv_cache --enable_context_fmha --max_input_len 4096 --max_batch_size 128 --strongly_typed --enable_fp8 --fp8_kv_cache --world_size 4 --tp_size 4 --parallel_build
The fp8 doesn't seem to be the issue given the error happened the same way across fp8/non fp8. Please keep in mind the version is still month old, but I did not see updates in the newer updates that would suggest this issue has been fixed so I just used the old engine and TRTLLM/Triton images I had built a month ago, based on https://github.com/NVIDIA/TensorRT-LLM/commit/6755a3f077bc39d51b74bbcb50403f54663cc8dc
@byshiue @juney-nvidia @jdemouth-nvidia
Update on this. Tried TP2, and no longer get any "illegal memory access" errors. However, when end_id = 2 is supplied from client side, the system will freeze indefinitely at high context + large batchsizes (20).
Removing end_id made it work, I tried 500 requests at 20 in parallel and none of it froze.
I can make a separate issue for end_id but it may also have to do with the older version that I built the engine with, but can confirm TP2 doesn't have the same behavior as TP4 in terms of memory access errors (at least not explicitly).
I used 4TP when deploying the 70B model. This NCCL error occurred on two of the PODs. This error does not occur on every POD. This error occurred after TRT-LLM was executed normally for a period of time.
TRT version: commit 4de32a86ae92bc49a7ec17c00ec2f2d03663c198 Execution env: 4*A100 80GB PCIE ERROR LOG: