Open SWHL opened 1 year ago
You can try to run the same shape 10 times and discard the time from the first run. Your number should be comparable to the static ones. If you keep changing the shape for each run, a lot of cached data will be invalidated and rebuilt.
I have the same problems when dealing with recognition model of PaddleOCR, due to dynamic shapes [-1, 3, 48, -1]. My suggestion is to warmup the model before doing inference, here snippet of model warmups
...
def model_warmup(self, batch_size: int = 1, min_size: int = 300, max_size: int = 1500):
"""
Recognition model have input size: [-1, 3, 48, -1]
ONNXRuntime with CUDA support is not performing well with arbitrary input size
So we need to warmup the model with arbitrary input size
"""
log.info("Warming up model...")
for i in tqdm(range(min_size, max_size), desc="Warming up model"):
dummy_input = np.random.randn(batch_size, 3, 48, i).astype(np.float32)
self.recog_session.run([self.recog_output_name], {self.recog_input_name: dummy_input})
log.info("Model warmup completed")
...
You can try to run the same shape 10 times and discard the time from the first run. Your number should be comparable to the static ones. If you keep changing the shape for each run, a lot of cached data will be invalidated and rebuilt.
@ytaous How many cache will ORT preserve for each model? For example, I have one model and inference 10 times with different input shapes. Will ORT preserve the last one shape cache, or last N shape cache? Or This is decided by another algorithm?
Describe the issue
To reproduce
The result:
Urgency
No response
Platform
Linux
OS Version
Ubuntu
ONNX Runtime Installation
Released Package
ONNX Runtime Version or Commit ID
1.12.1
ONNX Runtime API
Python
Architecture
X64
Execution Provider
CUDA
Execution Provider Library Version
CUDA 11.2
Model File
OCRv3_det_infer.zip
Is this a quantized model?
No