Inference of dla34_deform_576_768.onnx in tensorrt

[mamadouDembele]

Hi

I'm try to run dla34_deform_576_768.onnx model in tensorrt python. I got this error "[TRT] [E] 1: [runner.cpp::executeMyelinGraph::715] Error Code 1: Myelin ([myelinGraphExecute] Called without resolved dynamic shapes.)" .It's seems that the output shapes are dynamic (not fixed as usual). This is my first time dealing with this in tensorrt. I can't really find any resources on the internet on how to deal with this. I would like to know if you have already run the 3D detection model on tensorrt. If so, how did you manage this? If not, can you provide a model with a static shape model?

[Owen-Liuyuxuan]

Could you inform me of the TensorRT version and the Cuda version you are using?

[Owen-Liuyuxuan]

Mono3D is essentially doomed to have dynamic output unless you would like to decode output from the dense feature map.

But my warning and probably my workflow is different.

Best solution:

• Separate the operation on TensorRT/DLA and GPU. Let TensorRT or Jetson DLA perform the computation of the network, and transfer the result to GPU, then apply a standalone GPU section (using Onnx model or CUDA code) to decode the results from the dense feature map.
  • In this case, I may need to provide two Onnx models and a more complicated inference pipeline.

Fast Solution:

• Run symbolic inference on the current model:
  • Download this official synbofile to local. https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py
  • Optimize the provided model first. python3 -m onnxruntime.transformers.optimizer --input=dla34_deform_576_768.onnx --output=optimized.onnx
  • Run symbolic scripts python3 ./symbolic_shape_infer.py --input ./optimized.onnx --output ./output.onnx
  • Then you can use the output.onnx for TensorRT inference in this repo.
• The warm-up may be a little bit slow. Use one crowded images to warm up the inference then it will be very fast later.