Not giving correct output

[makaveli10]

I tried to test the generated engine on sample images in the yolov5 repo and I dont think the outputs are same for the trt engine and yolov5. Can you confirm? If you inference on images can you share the code? Precisely, the output is different in TRT and yolov5. Output shape of yolo for zidane.jpg: torch.Size([1, 15120, 85]) output shape of trt after reshaping : [1, 1200, 85]).

Below is the script that I used. Also if you inferenced on an image and got the correct output. I think it would be wonderful if you can share that! trt_inf.py.zip

Also, I checked the output on image of zeros in main.py and that doesnt match!!!

[TrojanXu]

Fixed with latest commit. the max element-wise diff is 0.0017089844 using bus.jpg. You can verify the result with other images as well.

[makaveli10]

@TrojanXu I am getting segmentation fault with the updated code I dont know what changed below is the terminal out:

%756 : Tensor = onnx::Shape(%500)
%757 : Tensor = onnx::Constant[value={2}]()
%758 : Long() = onnx::Gather[axis=0](%756, %757) # /home/yolo.py:28:0
%759 : Tensor = onnx::Shape(%500)
%760 : Tensor = onnx::Constant[value={3}]()
%761 : Long() = onnx::Gather[axis=0](%759, %760) # /home/yolo.py:28:0
%762 : Long() = onnx::Constant[value={3}]()
%763 : Long() = onnx::Constant[value={85}]()
%764 : Tensor = onnx::Unsqueeze[axes=[0]](%755)
%765 : Tensor = onnx::Unsqueeze[axes=[0]](%762)
%766 : Tensor = onnx::Unsqueeze[axes=[0]](%763)
%767 : Tensor = onnx::Unsqueeze[axes=[0]](%758)
%768 : Tensor = onnx::Unsqueeze[axes=[0]](%761)
%769 : Tensor = onnx::Concat[axis=0](%764, %765, %766, %767, %768)
%770 : Float(1, 3, 85, 20, 16) = onnx::Reshape(%500, %769) # /home/yolo.py:29:0
%771 : Float(1, 3, 20, 16, 85) = onnx::Transpose[perm=[0, 1, 3, 4, 2]](%770) # /home/yolo.py:29:0
%772 : Tensor = onnx::Unsqueeze[axes=[0]](%758)
%773 : Tensor = onnx::ConstantOfShape[value={1}](%772)
%774 : Tensor = onnx::NonZero(%773)
%775 : Tensor = onnx::Transpose[perm=[1, 0]](%774)
%776 : Tensor = onnx::Squeeze[axes=[1]](%775)
%777 : Long(20) = onnx::Cast[to=7](%776) # /home/yolo.py:47:0
%778 : Tensor = onnx::Unsqueeze[axes=[0]](%761)
%779 : Tensor = onnx::ConstantOfShape[value={1}](%778)
%780 : Tensor = onnx::NonZero(%779)
%781 : Tensor = onnx::Transpose[perm=[1, 0]](%780)
%782 : Tensor = onnx::Squeeze[axes=[1]](%781)
%783 : Long(16) = onnx::Cast[to=7](%782) # /home/yolo.py:47:0
%784 : Tensor = onnx::Constant[value={-1}]()
%785 : Tensor = onnx::Reshape(%777, %784)
%786 : Tensor = onnx::Constant[value={-1}]()
%787 : Tensor = onnx::Reshape(%783, %786)
%788 : Tensor = onnx::Shape(%785)
%789 : Tensor = onnx::Shape(%787)
%790 : Tensor = onnx::Concat[axis=0](%788, %789)
%791 : Tensor = onnx::Constant[value={1}]()
%792 : Tensor = onnx::Concat[axis=0](%788, %791)
%793 : Tensor = onnx::Reshape(%785, %792)
%794 : Tensor = onnx::Expand(%793, %790)
%795 : Tensor = onnx::Constant[value={1}]()
%796 : Tensor = onnx::Concat[axis=0](%795, %789)
%797 : Tensor = onnx::Reshape(%787, %796)
%798 : Tensor = onnx::Expand(%797, %790)
%799 : Tensor = onnx::Unsqueeze[axes=[2]](%798)
%800 : Tensor = onnx::Unsqueeze[axes=[2]](%794)
%801 : Long(20, 16, 2) = onnx::Concat[axis=2](%799, %800) # /home/yolo.py:48:0
%802 : Long() = onnx::Constant[value={1}]()
%803 : Long() = onnx::Constant[value={1}]()
%804 : Long() = onnx::Constant[value={2}]()
%805 : Tensor = onnx::Unsqueeze[axes=[0]](%802)
%806 : Tensor = onnx::Unsqueeze[axes=[0]](%803)
%807 : Tensor = onnx::Unsqueeze[axes=[0]](%758)
%808 : Tensor = onnx::Unsqueeze[axes=[0]](%761)
%809 : Tensor = onnx::Unsqueeze[axes=[0]](%804)
%810 : Tensor = onnx::Concat[axis=0](%805, %806, %807, %808, %809)
%811 : Long(1, 1, 20, 16, 2) = onnx::Reshape(%801, %810) # /home/yolo.py:48:0
%812 : Float(1, 1, 20, 16, 2) = onnx::Cast[to=1](%811) # /home/yolo.py:48:0
%813 : Float(1, 1, 20, 16, 2) = onnx::Cast[to=1](%812) # /home/yolo.py:33:0
%814 : Float(1, 3, 20, 16, 85) = onnx::Sigmoid(%771) # /home/yolo.py:35:0
%815 : Tensor = onnx::Constant[value={4}]()
%816 : Tensor = onnx::Constant[value={0}]()
%817 : Tensor = onnx::Constant[value={2}]()
%818 : Tensor = onnx::Constant[value={1}]()
%819 : Float(1, 3, 20, 16, 2) = onnx::Slice(%814, %816, %817, %815, %818) # /home/yolo.py:38:0
%820 : Float() = onnx::Constant[value={2}]()
%821 : Float(1, 3, 20, 16, 2) = onnx::Mul(%819, %820)
%822 : Float() = onnx::Constant[value={0.5}]()
%823 : Float(1, 3, 20, 16, 2) = onnx::Sub(%821, %822)
%824 : Float(1, 1, 20, 16, 2) = onnx::Cast[to=1](%813) # /home/yolo.py:38:0
%825 : Float(1, 3, 20, 16, 2) = onnx::Add(%823, %824) # /home/yolo.py:38:0
%826 : Float() = onnx::Constant[value={32}]()
%827 : Float(1, 3, 20, 16, 2) = onnx::Mul(%825, %826)
%828 : Tensor = onnx::Constant[value={4}]()
%829 : Tensor = onnx::Constant[value={2}]()
%830 : Tensor = onnx::Constant[value={4}]()
%831 : Tensor = onnx::Constant[value={1}]()
%832 : Float(1, 3, 20, 16, 2) = onnx::Slice(%814, %829, %830, %828, %831) # /home/yolo.py:39:0
%833 : Float() = onnx::Constant[value={2}]()
%834 : Float(1, 3, 20, 16, 2) = onnx::Mul(%832, %833)
%835 : Float() = onnx::Constant[value={2}]()
%836 : Float(1, 3, 20, 16, 2) = onnx::Pow(%834, %835) # /home/yolo.py:39:0
%837 : Tensor = onnx::Constant[value={2}]()
%838 : Float(1, 3, 1, 1, 2) = onnx::Gather[axis=0](%model.22.anchor_grid, %837) # /home/yolo.py:39:0
%839 : Float(1, 3, 20, 16, 2) = onnx::Mul(%836, %838) # /home/yolo.py:39:0
%840 : Float(1, 3, 20, 16, 2) = onnx::Cast[to=1](%839) # /home/yolo.py:40:0
%841 : Tensor = onnx::Constant[value={4}]()
%842 : Tensor = onnx::Constant[value={4}]()
%843 : Tensor = onnx::Constant[value={9223372036854775807}]()
%844 : Tensor = onnx::Constant[value={1}]()
%845 : Float(1, 3, 20, 16, 81) = onnx::Slice(%814, %842, %843, %841, %844) # /home/yolo.py:40:0
%846 : Float(1, 3, 20, 16, 81) = onnx::Cast[to=1](%845) # /home/yolo.py:40:0
%847 : Float(1, 3, 20, 16, 85) = onnx::Concat[axis=-1](%827, %840, %846) # /home/yolo.py:40:0
%848 : Long() = onnx::Constant[value={-1}]()
%849 : Long() = onnx::Constant[value={85}]()
%850 : Tensor = onnx::Unsqueeze[axes=[0]](%755)
%851 : Tensor = onnx::Unsqueeze[axes=[0]](%848)
%852 : Tensor = onnx::Unsqueeze[axes=[0]](%849)
%853 : Tensor = onnx::Concat[axis=0](%850, %851, %852)
%854 : Float(1, 960, 85) = onnx::Reshape(%847, %853) # /home/yolo.py:41:0
%model/22 : Float(1, 20160, 85) = onnx::Concat[axis=1](%650, %752, %854) # /home/yolo.py:43:0
return (%model/22)

Segmentation fault (core dumped)

This is the pip list output to check the package version:

    appdirs           1.4.3   
attrs             19.3.0  
Automat           20.2.0  
constantly        15.1.0  
construct         2.10.56 
cycler            0.10.0  
Cython            0.29.20 
decorator         4.4.2   
hyperlink         19.0.0  
idna              2.9     
incremental       17.5.0  
kiwisolver        1.2.0   
Mako              1.1.2   
MarkupSafe        1.1.1   
matplotlib        3.2.2   
numpy             1.17.0  
onnx              1.6.0   
onnx-simplifier   0.2.9   
onnxruntime       1.3.0   
opencv-python     4.2.0.34
Pillow            7.0.0   
pip               20.0.2  
ply               3.11    
protobuf          3.12.2  
pyasn1            0.4.8   
pycocotools       2.0     
pycryptodomex     3.9.7   
pycuda            2019.1.2
PyHamcrest        2.0.2   
pyparsing         2.4.7   
pysmi             0.3.4   
pysnmp            4.4.12  
python-dateutil   2.8.1   
pytools           2020.1  
PyYAML            5.3.1   
scipy             1.4.1   
setuptools        45.2.0  
six               1.14.0  
tensor            0.3.6   
tensorrt          7.0.0.11
torch             1.4.0   
torchvision       0.5.0   
tqdm              4.46.1  
Twisted           20.3.0  
typing-extensions 3.7.4.2 
wheel             0.34.2  
zope.interface    5.1.0  

If you could help out it would wonderful! Thanks

[makaveli10]

While debugging I figured out that this seg fault comes in onnx-simplify function! I mean onnx is exported succesfully. (if that helps). Precisely on line no 56:

model_simp, check = simplify(model)

[TrojanXu]

I currently have no idea why this causes a segfault in onnx-sim from your pip list. Could you try 'python -m onnxsim input_onnx_model output_onnx_model' directly? You can also try to upgrade your onnx to 1.7.0 (I also tried this version, and it worked in my env). BTW, I'm using pytorch:20.01-py3 docker image from ngc.nvidia.com

[makaveli10]

root@28377c0239a0:/home# python -m onnxsim yolov5_1.onnx yolov5_1.onnx Simplifying... Checking 0/3... Checking 1/3... Checking 2/3... Ok!

Yeah this works Onto checking the output! Thanks