search

triton-inference-server / tensorrtllm_backend

The Triton TensorRT-LLM Backend
Apache License 2.0
652 stars 93 forks source link

ERROR: Failed to create instance: unexpected error when creating modelInstanceState #328

Open ken2190 opened 7 months ago

ken2190 commented 7 months ago

System Info

Who can help?

No response

Information

Reproduction

TensorRT-LLM (cloned from https://github.com/NVIDIA/TensorRT-LLM, git checkout v0.5.0)
tensorrtllm_backend (cloned from https://github.com/triton-inference-server/tensorrtllm_backend, git checkout v0.5.0)

mkdir -p /home/ubuntu/DATA/tensorrtllm_backend/engines
docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864  \
                --gpus=all \
                --volume  /home/ubuntu/DATA/llama_hf/CodeLlama-7b-Instruct:/CodeLlama-7b-Instruct \
                --volume  /home/ubuntu/DATA/tensorrtllm_backend/engines:/engines \
                --volume /home/ubuntu/DATA:/code/tensorrt_llm \
                --workdir /code/tensorrt_llm/TensorRT-LLM \
                --hostname t4-release \
                --name tensorrt_llm-release-ubuntu \
                --tmpfs /tmp:exec \
                tensorrt_llm/release:latest

python3 examples/llama/build.py --model_dir /CodeLlama-7b-Instruct/ \
                --dtype float16 \
                --use_gpt_attention_plugin float16 \
                --use_inflight_batching \
                --paged_kv_cache \
                --remove_input_padding \
                --use_gemm_plugin float16 \
                --output_dir /engines/1-gpu/ \
                --world_size 1

Model built without problem

docker run --rm -it --net host --shm-size=2g \
    --ulimit memlock=-1 --ulimit stack=67108864 --gpus all \
    -v /home/ubuntu/DATA/tensorrtllm_backend:/tensorrtllm_backend \
    -v /home/ubuntu/DATA/llama_hf/CodeLlama-7b-Instruct:/CodeLlama-7b-Instruct \
    -v /home/ubuntu/DATA/tensorrtllm_backend/engines:/engines \
    nvcr.io/nvidia/tritonserver:23.10-trtllm-python-py3

cp -R /tensorrtllm_backend/all_models/inflight_batcher_llm /opt/tritonserver/.
cp /engines/1-gpu/* inflight_batcher_llm/tensorrt_llm/1/

HF_LLAMA_MODEL=/CodeLlama-7b-Instruct
ENGINE_DIR=/engines/1-gpu

python3 /tensorrtllm_backend/tools/fill_template.py -i inflight_batcher_llm/preprocessing/config.pbtxt tokenizer_dir:${HF_LLAMA_MODEL},tokenizer_type:llama,triton_max_batch_size:64,preprocessing_instance_count:1
python3 /tensorrtllm_backend/tools/fill_template.py -i inflight_batcher_llm/postprocessing/config.pbtxt tokenizer_dir:${HF_LLAMA_MODEL},tokenizer_type:llama,triton_max_batch_size:64,postprocessing_instance_count:1
python3 /tensorrtllm_backend/tools/fill_template.py -i inflight_batcher_llm/ensemble/config.pbtxt triton_max_batch_size:64
python3 /tensorrtllm_backend/tools/fill_template.py -i inflight_batcher_llm/tensorrt_llm/config.pbtxt triton_max_batch_size:64,decoupled_mode:False,max_beam_width:1,engine_dir:${ENGINE_DIR},max_tokens_in_paged_kv_cache:2560,max_attention_window_size:2560,kv_cache_free_gpu_mem_fraction:0.5,exclude_input_in_output:True,enable_kv_cache_reuse:False,batching_strategy:inflight_batching,max_queue_delay_microseconds:600,batch_scheduler_policy:guaranteed_no_evict

root@t4:/opt/tritonserver# pip install sentencepiece
root@t4:/opt/tritonserver# pip install protobuf

root@t4:/opt/tritonserver# python3 /tensorrtllm_backend/scripts/launch_triton_server.py --world_size=1 --model_repo=/opt/tritonserver/inflight_batcher_llm/

Expected behavior

I0922 23:28:40.351809 1 grpc_server.cc:2451] Started GRPCInferenceService at 0.0.0.0:8001
I0922 23:28:40.352017 1 http_server.cc:3558] Started HTTPService at 0.0.0.0:8000
I0922 23:28:40.395611 1 http_server.cc:187] Started Metrics Service at 0.0.0.0:8002

actual behavior

root@t4:/opt/tritonserver# pip install sentencepiece
root@t4:/opt/tritonserver# pip install protobuf

root@t4:/opt/tritonserver# python3 /tensorrtllm_backend/scripts/launch_triton_server.py --world_size=1 --model_repo=/opt/triton
server/inflight_batcher_llm/
root@t4:/opt/tritonserver# [TensorRT-LLM][WARNING] max_num_sequences is not specified, will be set to the TRT engine max_batch_size
[TensorRT-LLM][WARNING] enable_trt_overlap is not specified, will be set to true
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be number, but is null
[TensorRT-LLM][WARNING] Optional value for parameter max_num_tokens will not be set.
[TensorRT-LLM][INFO] Initializing MPI with thread mode 1
[TensorRT-LLM][INFO] MPI size: 1, rank: 0
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 6
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 3
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 1
Cleaning up...
Cleaning up...
I0205 17:02:21.226812 416 pinned_memory_manager.cc:241] Pinned memory pool is created at '0x10018000000' with size 268435456
I0205 17:02:21.228151 416 cuda_memory_manager.cc:107] CUDA memory pool is created on device 0 with size 67108864
I0205 17:02:21.234199 416 model_lifecycle.cc:461] loading: tensorrt_llm:1
I0205 17:02:21.234321 416 model_lifecycle.cc:461] loading: postprocessing:1
I0205 17:02:21.234458 416 model_lifecycle.cc:461] loading: preprocessing:1
I0205 17:02:21.365042 416 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: postprocessing_0_0 (CPU device 0)
I0205 17:02:21.365285 416 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: preprocessing_0_0 (CPU device 0)
I0205 17:02:22.409739 416 model_lifecycle.cc:818] successfully loaded 'postprocessing'
I0205 17:02:25.260513 416 model_lifecycle.cc:818] successfully loaded 'preprocessing'
E0205 17:02:40.250190 416 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: [TensorRT-LLM][ERROR] CUDA runtime error in cudaDeviceGetDefaultMemPool(&memPool, device): operation not supported (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/bufferManager.cpp:170)
1       0x7f83fe81e045 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x36045) [0x7f83fe81e045]
2       0x7f83fe873925 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8b925) [0x7f83fe873925]
3       0x7f83fe8747bf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8c7bf) [0x7f83fe8747bf]
4       0x7f83fe8d5545 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0xed545) [0x7f83fe8d5545]
5       0x7f83fe84ef4e /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x66f4e) [0x7f83fe84ef4e]
6       0x7f83fe83ec0c /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x56c0c) [0x7f83fe83ec0c]
7       0x7f83fe8395f5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x515f5) [0x7f83fe8395f5]
8       0x7f83fe8374db /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x4f4db) [0x7f83fe8374db]
9       0x7f83fe81b182 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x33182) [0x7f83fe81b182]
10      0x7f83fe81b235 TRITONBACKEND_ModelInstanceInitialize + 101
11      0x7f845419aa86 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a4a86) [0x7f845419aa86]
12      0x7f845419bcc6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a5cc6) [0x7f845419bcc6]
13      0x7f845417ec15 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x188c15) [0x7f845417ec15]
14      0x7f845417f256 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x189256) [0x7f845417f256]
15      0x7f845418b27d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19527d) [0x7f845418b27d]                             16      0x7f84537f9ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f84537f9ee8]
17      0x7f845417597b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x17f97b) [0x7f845417597b]
18      0x7f8454185695 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18f695) [0x7f8454185695]
19      0x7f845418a50b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19450b) [0x7f845418a50b]
20      0x7f8454273610 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x27d610) [0x7f8454273610]
21      0x7f8454276d03 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x280d03) [0x7f8454276d03]
22      0x7f84543c38b2 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3cd8b2) [0x7f84543c38b2]
23      0x7f8453a64253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f8453a64253]
24      0x7f84537f4ac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f84537f4ac3]
25      0x7f8453885bf4 clone + 68
E0205 17:02:40.250573 416 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 1: Internal: unexpected error when crea
ting modelInstanceState: [TensorRT-LLM][ERROR] CUDA runtime error in cudaDeviceGetDefaultMemPool(&memPool, device): operation not supported (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/bufferManager.cpp:170)
1       0x7f83fe81e045 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x36045) [0x7f83fe81e045]
2       0x7f83fe873925 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8b925) [0x7f83fe873925]
3       0x7f83fe8747bf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8c7bf) [0x7f83fe8747bf]
4       0x7f83fe8d5545 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0xed545) [0x7f83fe8d5545]
5       0x7f83fe84ef4e /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x66f4e) [0x7f83fe84ef4e]
6       0x7f83fe83ec0c /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x56c0c) [0x7f83fe83ec0c]
7       0x7f83fe8395f5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x515f5) [0x7f83fe8395f5]
8       0x7f83fe8374db /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x4f4db) [0x7f83fe8374db]
9       0x7f83fe81b182 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x33182) [0x7f83fe81b182]
10      0x7f83fe81b235 TRITONBACKEND_ModelInstanceInitialize + 101
11      0x7f845419aa86 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a4a86) [0x7f845419aa86]                    [183/495]
12      0x7f845419bcc6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a5cc6) [0x7f845419bcc6]
13      0x7f845417ec15 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x188c15) [0x7f845417ec15]
14      0x7f845417f256 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x189256) [0x7f845417f256]
15      0x7f845418b27d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19527d) [0x7f845418b27d]
16      0x7f84537f9ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f84537f9ee8]
17      0x7f845417597b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x17f97b) [0x7f845417597b]
18      0x7f8454185695 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18f695) [0x7f8454185695]
19      0x7f845418a50b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19450b) [0x7f845418a50b]
20      0x7f8454273610 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x27d610) [0x7f8454273610]
21      0x7f8454276d03 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x280d03) [0x7f8454276d03]
22      0x7f84543c38b2 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3cd8b2) [0x7f84543c38b2]
23      0x7f8453a64253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f8453a64253]
24      0x7f84537f4ac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f84537f4ac3]
25      0x7f8453885bf4 clone + 68
I0205 17:02:40.250673 416 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E0205 17:02:40.250868 416 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'tensorrt_llm' whic
h has no loaded version. Model 'tensorrt_llm' loading failed with error: version 1 is at UNAVAILABLE state: Internal: unexpecte
d error when creating modelInstanceState: [TensorRT-LLM][ERROR] CUDA runtime error in cudaDeviceGetDefaultMemPool(&memPool, device): operation not supported (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/bufferManager.cpp:170)
1       0x7f83fe81e045 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x36045) [0x7f83fe81e045]
2       0x7f83fe873925 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8b925) [0x7f83fe873925]
3       0x7f83fe8747bf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8c7bf) [0x7f83fe8747bf]
4       0x7f83fe8d5545 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0xed545) [0x7f83fe8d5545]
5       0x7f83fe84ef4e /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x66f4e) [0x7f83fe84ef4e]
6       0x7f83fe83ec0c /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x56c0c) [0x7f83fe83ec0c]
7       0x7f83fe8395f5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x515f5) [0x7f83fe8395f5]
8       0x7f83fe8374db /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x4f4db) [0x7f83fe8374db]
9       0x7f83fe81b182 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x33182) [0x7f83fe81b182]
10      0x7f83fe81b235 TRITONBACKEND_ModelInstanceInitialize + 101
11      0x7f845419aa86 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a4a86) [0x7f845419aa86]
12      0x7f845419bcc6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a5cc6) [0x7f845419bcc6]
13      0x7f845417ec15 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x188c15) [0x7f845417ec15]
14      0x7f845417f256 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x189256) [0x7f845417f256]
15      0x7f845418b27d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19527d) [0x7f845418b27d]
16      0x7f84537f9ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f84537f9ee8]
17      0x7f845417597b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x17f97b) [0x7f845417597b]
18      0x7f8454185695 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18f695) [0x7f8454185695]
19      0x7f845418a50b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19450b) [0x7f845418a50b]
20      0x7f8454273610 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x27d610) [0x7f8454273610]
21      0x7f8454276d03 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x280d03) [0x7f8454276d03]
22      0x7f84543c38b2 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3cd8b2) [0x7f84543c38b2]
23      0x7f8453a64253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f8453a64253]
24      0x7f84537f4ac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f84537f4ac3]
25      0x7f8453885bf4 clone + 68;
I0205 17:02:40.251051 416 server.cc:592]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0205 17:02:40.251131 416 server.cc:619]
+-------------+-----------------------------------------------------------------+----------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
-----------------------+
| Backend     | Path                                                            | Config

                       |
+-------------+-----------------------------------------------------------------+----------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
-----------------------+
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","b
ackend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}}
                       |
| python      | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"false","b
ackend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","shm-region-prefix-name":"prefix0_","default
-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+----------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
-----------------------+

I0205 17:02:40.251331 416 server.cc:662]
+----------------+---------+---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
-------------------------------+
| Model          | Version | Status

                               |
+----------------+---------+---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
-------------------------------+
| postprocessing | 1       | READY

                               |
| preprocessing  | 1       | READY

                               |
| tensorrt_llm   | 1       | UNAVAILABLE: Internal: unexpected error when creating modelInstanceState: [TensorRT-LLM][ERROR] CU
DA runtime error in cudaDeviceGetDefaultMemPool(&memPool, device): operation not supported (/app/tensorrt_llm/cpp/tensorrt_llm/
runtime/bufferManager.cpp:170) |
|                |         | 1       0x7f83fe81e045 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x36045) [
0x7f83fe81e045]
                               |
|                |         | 2       0x7f83fe873925 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8b925) [
0x7f83fe873925]
                               |
|                |         | 3       0x7f83fe8747bf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x8c7bf) [
0x7f83fe8747bf]
                               |
|                |         | 4       0x7f83fe8d5545 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0xed545) [
0x7f83fe8d5545]
                               |
|                |         | 5       0x7f83fe84ef4e /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x66f4e) [
0x7f83fe84ef4e]
                               |
|                |         | 6       0x7f83fe83ec0c /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x56c0c) [
0x7f83fe83ec0c]
                               |
|                |         | 7       0x7f83fe8395f5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x515f5) [
0x7f83fe8395f5]
                               |
|                |         | 8       0x7f83fe8374db /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x4f4db) [
0x7f83fe8374db]
                               |
|                |         | 9       0x7f83fe81b182 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x33182) [
0x7f83fe81b182]
                               |
|                |         | 10      0x7f83fe81b235 TRITONBACKEND_ModelInstanceInitialize + 101

                               |
|                |         | 11      0x7f845419aa86 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a4a86) [0x7f845419aa86]

                               |
|                |         | 12      0x7f845419bcc6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a5cc6) [0x7f845419bcc6]

                               |
|                |         | 13      0x7f845417ec15 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x188c15) [0x7f845417ec15]

                               |
|                |         | 14      0x7f845417f256 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x189256) [0x7f845417f256]

                               |
|                |         | 15      0x7f845418b27d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19527d) [0x7f845418b27d]

                               |
|                |         | 16      0x7f84537f9ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7f84537f9ee8]

                               |
|                |         | 17      0x7f845417597b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x17f97b) [0x7f845417597b]

                               |
|                |         | 18      0x7f8454185695 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18f695) [0x7f8454185695]

                               |
|                |         | 19      0x7f845418a50b /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19450b) [0x7f845418a50b]

                               |
|                |         | 20      0x7f8454273610 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x27d610) [0x7f8454273610]

                               |
|                |         | 21      0x7f8454276d03 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x280d03) [0x7f8454276d03]

                               |
|                |         | 22      0x7f84543c38b2 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3cd8b2) [0x7f845[30/495]

                               |
|                |         | 23      0x7f8453a64253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f8453a64253]

                               |
|                |         | 24      0x7f84537f4ac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f84537f4ac3]

                               |
|                |         | 25      0x7f8453885bf4 clone + 68

                               |
+----------------+---------+---------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------
-------------------------------+

I0205 17:02:40.271435 416 metrics.cc:817] Collecting metrics for GPU 0: GRID V100S-32Q
I0205 17:02:40.271838 416 metrics.cc:710] Collecting CPU metrics
I0205 17:02:40.272083 416 tritonserver.cc:2458]
+----------------------------------+-------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------+
| Option                           | Value
                                                                                                                      |
+----------------------------------+-------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton
                                                                                                                      |
| server_version                   | 2.39.0
                                                                                                                      |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_poli
cy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0]         | /opt/tritonserver/inflight_batcher_llm/
                                                                                                                      |
| model_control_mode               | MODE_NONE
                                                                                                                      |
| strict_model_config              | 1
                                                                                                                      |
| rate_limit                       | OFF
                                                                                                                      |
| pinned_memory_pool_byte_size     | 268435456
                                                                                                                      |
| cuda_memory_pool_byte_size{0}    | 67108864
                                                                                                                      |
| min_supported_compute_capability | 6.0
                                                                                                                      |
| strict_readiness                 | 1
                                                                                                                      |
| exit_timeout                     | 30
                                                                                                                      |
| cache_enabled                    | 0
                                                                                                                      |
+----------------------------------+-------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------+

I0205 17:02:40.272122 416 server.cc:293] Waiting for in-flight requests to complete.
I0205 17:02:40.272151 416 server.cc:309] Timeout 30: Found 0 model versions that have in-flight inferences
I0205 17:02:40.272348 416 server.cc:324] All models are stopped, unloading models
I0205 17:02:40.272370 416 server.cc:331] Timeout 30: Found 2 live models and 0 in-flight non-inference requests
I0205 17:02:41.272520 416 server.cc:331] Timeout 29: Found 2 live models and 0 in-flight non-inference requests
W0205 17:02:41.276802 416 metrics.cc:582] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0205 17:02:41.276858 416 metrics.cc:600] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0205 17:02:41.276883 416 metrics.cc:624] Unable to get energy consumption for GPU 0. Status:Success, value:0
I0205 17:02:41.641130 416 model_lifecycle.cc:603] successfully unloaded 'postprocessing' version 1
I0205 17:02:42.046922 416 model_lifecycle.cc:603] successfully unloaded 'preprocessing' version 1
I0205 17:02:42.272760 416 server.cc:331] Timeout 28: Found 0 live models and 0 in-flight non-inference requests
W0205 17:02:42.277527 416 metrics.cc:582] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0205 17:02:42.277595 416 metrics.cc:600] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0205 17:02:42.277630 416 metrics.cc:624] Unable to get energy consumption for GPU 0. Status:Success, value:0
error: creating server: Internal - failed to load all models
W0205 17:02:43.279857 416 metrics.cc:582] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0205 17:02:43.279967 416 metrics.cc:600] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0205 17:02:43.279983 416 metrics.cc:624] Unable to get energy consumption for GPU 0. Status:Success, value:0
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[55476,1],0]
  Exit code:    1
--------------------------------------------------------------------------

additional notes

@byshiue @schetlur-nv @juney-nvidia Could you take a look

byshiue commented 6 months ago

Could you try latest main branch?

tobernat commented 6 months ago

I get the exact same error when trying to replicate the following tutorial with version 0.5.0: https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/ I tried with the latest main branch, but still got the error [TensorRT-LLM][ERROR] CUDA runtime error in cudaDeviceGetDefaultMemPool(&memPool, device): operation not supported

Has the issue been resolved?

schetlur-nv commented 6 months ago

@tobernat I see on https://github.com/triton-inference-server/tensorrtllm_backend/issues/363 that you were able to resolve this. Is there anything more that needs to be done? @ken2190 can you please refer to the fix posted in issue 363 linked above?

tobernat commented 5 months ago

In our case (see #363) it was solved by setting vGPU plugin parameters in VMware: https://docs.nvidia.com/grid/13.0/grid-vgpu-user-guide/index.html#setting-vgpu-plugin-parameters-on-vmware-vsphere see also here: https://kb.vmware.com/s/article/2142307

renareke commented 5 months ago

In our case (see #363) it was solved by setting vGPU plugin parameters in VMware: https://docs.nvidia.com/grid/13.0/grid-vgpu-user-guide/index.html#setting-vgpu-plugin-parameters-on-vmware-vsphere see also here: https://kb.vmware.com/s/article/2142307 @tobernat What parameters where needed to set to get this issue resolved? I have the same issue with a L40s card.

Thanks for the reply!