Open wq9 opened 2 months ago
[6864]. created for tracking
I have the same problem. It is probably caused by the dali backend dali.fn.experimental.inputs.video API, because I have tried using the minimal code and the problem also happened.
version : 23.07
Description Triton uses over 100% of physical memory and freezes the server when using a decoupled dali model with a long video input.
Triton Information Docker
nvcr.io/nvidia/tritonserver:24.03-py3
To Reproduce
resize_384/1/dali.py
resize_384/config.pbtxt
client.py
Run triton server with
docker run -it --gpus=all --net=host --rm -v /home/ubuntu:/home/ubuntu nvcr.io/nvidia/tritonserver:24.03-py3 tritonserver --model-repository=/home/ubuntu/model_repository
, then runclient.py
with video of ~18min.Triton server memory usage grows to 75GB of RAM.![image](https://github.com/triton-inference-server/server/assets/3300313/3247b8a9-7660-4f90-86db-3bcce61aab39)
If using a longer video, RAM memory usage goes over 100% and the server freezes.
Expected behavior RAM usage should not go over 100% and should not depend on input video length.