Dynamic Shape performance

[aitss2017]

Describe the bug We want to compare static shape and dynamic shape performance difference for Bert model, but we measured similar number. Not sure it is correct.

Urgency Dynamic shape featute is key for NLP models, especially for bert series models

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.8.2003 (Core)
• ONNX Runtime installed from (source or binary): pip install onnxruntime
• ONNX Runtime version: 1.10
• Python version: 3.7
• Visual Studio version (if applicable): None
• GCC/Compiler version (if compiling from source): 7.3.1
• CUDA/cuDNN version: None
• GPU model and memory: None

To Reproduce

• Describe steps/code to reproduce the behavior.
• Attach the ONNX model to the issue (where applicable) to expedite investigation.

Expected behavior Some guide to measure dynamic and static shape performance difference

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here. If the issue is about a particular model, please share the model details as well to facilitate debugging.

[tianleiwu]

I think it is expected that you can get same performance for same input.

Dynamic shape only means the model is more flexible (can accept input of different shape). For BERT model with dynamic shape, we recommend to use transformer optimizer tool to get optimized model before inferencing.

[aitss2017]

Thanks for your comment, Tianle! We have leverage such transformer optimizer.

Can we say that ORT would measure same performance for dynamic and static shape for all models besides Bert? Such as detection and image super resolution models, which would request dynamic shape support.

If yes, then can you help me understand how ORT solve such dynamic shape support? Such as memory allocation, kernel caching, etc.

[stale[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.