Closed ghost closed 3 years ago
Tensorflow framework allocates some memory when the model is loaded but will dynamic allocate more (potentially a lot more) as it is processing inference requests. Do you have some evidence to expect memory usage to be constant with tensorflow savedmodel? Have you tried running the model directly using TF apis (for example by writing a python script to load and run)? Triton has a model analyzer project that can help you understand the memory requirements of your project: https://github.com/triton-inference-server/model_analyzer
Would converting the model to TensorRT first stop this?
With Triton 1.x we didn't see this behaviour with the same models
TensorFlow's behavior wrt memory allocation hasn't changed in a long time, so I don't know why you are seeing something different now. TensorRT models allocate their entire GPU memory requirement when they are loaded, so you would not have this problem of memory growing during inferencing... at least from the model. Memory usage could still grow if you were overloading the server and so many requests were queuing up, but this should be CPU memory and there are model configuration settings to limit the queue size (https://github.com/triton-inference-server/server/blob/master/docs/model_configuration.md#scheduling-and-batching).
With TensorRT you pick the maximum allowed batch size when you create the model and so memory required to support that will be allocated when the model is loaded.
So you don't know how your software behaves, and you don't know how long it has behaved this way i.e. how long is "long time" in software? Don't you mean: since version x.x?
We don't have a CPU problem the server has 28 cores. They never max out.
When we compile Efficientnet from the savedmodel
to TensorRT we get a lots of "Algorithm not found" errors.
Hmm, have you tried to load everything on a dgx?
The TensorRT problem was specific to a particular combination of TensorRT version and the CUDA driver version its being run on
Description
We load up a couple of 2080s with ~75 Tensorflow
.savemodel
modelsMemory usage is about 95%, then when inference starts this usage creps up to 99%. we then tested this with 1 model ~100MB and use your
perf_client
to do some load testing on different batch sizes and the memory eventually gets comepletly consumered.Triton logs have lots of this type of thing:
ending with something like this:
This behaviour keeps repeating. Inference takes minutes if we keep the batch size to 1 anything greater than that and it stops working completely.
Models are Efficientnet-b5. Can send you one of our models if you want to test this.
Triton Information Container 20.09
To Reproduce Load 75 of these models on a 2080 and try and do inference.
Expected behavior Memory is constant and inference is fast