-
**Issue Description:**
During a graceful shutdown of Triton Server, we've observed the following behavior:
- Triton Server is hosting both Model A and Model B.
- Model B can make calls to Model…
-
### System Info
arch - x86-64
gpu - rtx3070
docker image nvcr.io/nvidia/tritonserver:24.01-trtllm-python-py3
tensorRT-LLM-backend tag - 0.7.2
tensorRT-LLM tag - 0.7.1 (80bc07510ac4ddf13c0d76ad2…
-
**Description**
I was using Triton Server nvcr.io/nvidia/tritonserver:24.04-py3 on my local machine with Windows 10 via docker container. Ie installed latest Nvidia Driver 555.85, and docker containe…
-
**Is your feature request related to a problem? Please describe.**
I find the current docker image `xx.yy-py3` doesn't have commonly use data preprocessing libraries like huggingface transformers f…
HQ01 updated
4 months ago
-
https://github.com/triton-inference-server/redis_cache/blob/437e7e214ff446e6bc12febb44872547a1988fe1/src/redis_cache.cc#L220
Triton inference server is failing to write to Redis on versions past 4.…
-
Dear Developers:
I'm deploying a GPT model with triton-inference-server and fastertransformer_backend, following this tutorial: https://github.com/triton-inference-server/fastertransformer_backend/…
-
### System Info
I have searched the repo here and the main server repo but don't see any information on either a) support for Safetensors (many models are saved that way on HF) and also b) whether th…
-
I'm trying to use Triton to deploy baichuan2-13B inference under bf16 precision. The tritonserver can be started successfully, but when processing client request, it crashed.
- Use TensorRT-LLM v0…
-
### Proposal to improve performance
_No response_
### Report of performance regression
_No response_
### Misc discussion on performance
To reproduce vLLM's performance benchmark, please…
-
Hi there,
Thanks again for this library!
We're trying to convert a fine-tuned T5 model to ONNX and run it in Triton. We've managed to convert the model to ONNX and use the T5 notebook guide to…