Open gulldan opened 3 months ago
Tracking ticket: [DLIS-6397]
Hi @gulldan, compose.py
doesn't currently support the TensorRT-LLM backend (DLIS-6397).
You should be able to achieve something similar by using build.py
with:
--backend tensorrtllm:r24.04
--backend python:r24.04
--backend onnxruntime:r24.04
Let us know if this helps for your use case.
thank you.
i tried
./build.py --backend tensorrtllm:r24.04 --backend python:r24.04 --backend onnxruntime:r24.04 --enable-gpu --build-type Release --target-platform linux --endpoint grpc --endpoint http
but its failed build_log.txt
Host info Linux 6.5.0-35-generic #35-Ubuntu SMP PREEMPT_DYNAMIC Fri Apr 26 11:23:57 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Docker version 26.1.2, build 211e74b cmake version 3.28.4 python 3.11.6 GeForce RTX 4090 Driver Version: 550.54.15
Im trying https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/compose.md to build onnx+python+tensorrtllm backends.
1) as mention in doc i do
and it builds, but when i start triton server.
E0517 12:18:34.314931 164 model_lifecycle.cc:638] failed to load 'llama3_tensorrt_llm' version 1: Invalid argument: unable to find backend library for backend 'tensorrtllm', try specifying runtime on the mode configuration.
models with python, onnx loads correct.
How i can combine docker image for using both backends?
2)
failed, tensorrt not found