qianqing13579
commented
1 year ago Hello @CharlieL7 @umangyadav , We have many projects which need dynamic shape,and I have been developing dynamic shape of migraphx recently.My project is here. I want to share some ideas and the problems I encountered in the implementation. I hope we can solve the dynamic shape together.
The reasons why migraphx does not support dynamic shape can list as follows:
1. MIGraphX IR does not support dynamic models the following ops in MIGraphX IR can not support dynamic models
- reshape
- constantofshape
- slice
- expand
- range
- resize
- pad
- tile
the output shapes of thess ops are depend on the value of input tensor which are called data dependent in TVM relax(https://github.com/tlc-pack/relax/wiki/Relax-Architecture-Overview). These ops should be extended to support dynamic shape.
2. MIGraphX compilation needs shape many passes in compilation need shape information,such as memory coloring ,fuse ops,but dynamic models can not provide shape information in compilation.
3. gpu kernels can not run with different shapes MIGraphX JIT kernel can not run with different shapes, such as pointwise jit kernels. what's more, some ops are replaced with other ops,such as resize and lstm,which can cause ops do not support dynamic shape.
4. MIGraphX can not compute shape in runtime dynamic models need compute the output shapes of each op according to input shapes in runtime.
My solutions about the dynamic shape problems are as follows:
- extend MIGraphX IR.
- set a max shape in parse_onnx.The shapes can not exceed the max shape in runtime. we can do many optimizations in compilation with a max shape in dynamic models,such as memory coloring
- rewrite jit implementation kernel, such as resize kernel
- use a shape function that can compute shape in runtime
| My solutions can support many CV and NLP models with dynamic shape. | support mode | dynamic mode |
|---|---|---|
| ResNet50 | N，H，W | |
| InceptionV3 | N，H，W | |
| MobileNetV2 | N，H，W | |
| DenseNet | N，H，W | |
| MTCNN | N，H，W | |
| SSD-VGG16 | N，H，W | |
| RetinaNet | N，H，W | |
| RetinaFace | N，H，W | |
| YOLOV3 | N，H，W | |
| YOLOV5 | N，H，W | |
| DBNet | N，H，W | |
| FCN | N，H，W | |
| UNet | N，H，W | |
| CRNN-LSTM | H,W | |
| SVTR | H,W | |
| BERT | dynamic sequence | |
| Transformer | dynamic sequence | |
| GPT2 | dynamic sequence |
The problems in my solutions:
- The performance is poor. The shape related ops in my solutions are assigned to GPU which cause bad performance. These ops should be assigned to CPU to improve perf. The miopen conv performance in dynamic shape is poor.
- Can not support models if the input arguments of data dependent ops are model input params
A dynamic shape demo:
#include <migraphx/onnx.hpp>
#include <migraphx/gpu/target.hpp>
// Set a max shape
migraphx::onnx_options onnx_options;
onnx_options.map_input_dims["input"]={8,3,224,224};
// Load model
migraphx::program net= migraphx::parse_onnx("ResNet50.onnx",onnx_options);
// Compile
migraphx::compile_options options;
net.compile(migraphx::gpu::target{},options);
// Set different shapes in runtime
std::vector<std::vector<std::size_t>> inputShapes;
inputShapes.push_back({1,3,112,112});
inputShapes.push_back({2,3,224,224});
// Run inference with different shapes
for(int i=0;i<inputShapes.size();++i)
{
// Load and preprocess the input image to inputTensor with different shapes
...
// Run inference
std::vector<migraphx::argument> results = net.eval(inputTensor);
...
}
CharlieL7
lzd-1230
the following ops do not support dynamic shape in migraphx.
I hope that migraphx can fully support dynamic shape as soon as possible.Thanks