Open aftnix opened 1 year ago
Hi @aftnix - Have the encoder and decoder models been optimized using the offline transformer optimization tooling ? Tagging @tianleiwu to confirm if there are optimizations available for vision encoder-decoder models...
I took a quick look at the code, providers=["CPUExecutionProvider"]
means it is using CPU EP. Need change to providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
.
For CUDA EP, it also need add I/O Binding (see example here.) to make sure the encoder output stays in GPU memory. Current code will copy hidden state to CPU, then ORT will need copy it from CPU to GPU. That wastes time.
onnx_inputs["encoder_hidden_states"] = encoder_hidden_states.cpu().detach().numpy()
The transformer optimization tool might help. Try python -m onnxruntime.transformers.optimizer --input model.onnx --output optimized.onnx --model_type unet
to see whether it could help.
Describe the issue
I'm getting 3X slower inference on a fine-tuned huggingface model. The script :
Output :
Nvidia Driver and CUDA version :
NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.8
To reproduce
Urgency
No response
Platform
Linux
OS Version
root@bdbfc49cda2d:/workspace# uname -a Linux bdbfc49cda2d 4.14.318-167.530.amzn1.x86_64 #1 SMP Tue Jun 27 22:03:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
ONNX Runtime Installation
Released Package
ONNX Runtime Version or Commit ID
1.15.1
ONNX Runtime API
Python
Architecture
X64
Execution Provider
CUDA
Execution Provider Library Version
11.8
Model File
No response
Is this a quantized model?
No