search

Mandylove1993 / CUDA-FastBEV

TensorRT deploy and PTQ/QAT tools development for FastBEV, total time only need 6.9ms!!!
MIT License
233 stars 38 forks source link

TRTModel inference time exception on NVidia Orin #9

Closed LiuXuan66 closed 1 year ago

LiuXuan66 commented 1 year ago

Hi, I just use tensorrt 8.5.2 to export the provided onnx model to trt on Orin . However, the inference time of int8 resnet18 model reaches 60ms per frame, fp16 reaches 100ms . I try to find reasons but failed. (eg. set 'release' not 'debug',update the tensorrt)

LiuXuan66 commented 1 year ago

imagenrom：5.27702ms camera backbone:29.77021ms BEV vision transform :1.94419ms BEV fuse head: 17.78800ms BEV postprocess: 5.88835ms Total: 55.391ms

LiuXuan66 commented 1 year ago

I try again on another new Orin mechine. imagenrom：4.20195ms camera backbone:12.36874ms BEV vision transform :1.30166ms BEV fuse head: 69.65811ms BEV postprocess: 4.74912ms Total: 88.078ms

Mandylove1993 commented 1 year ago

I just test on 2080Ti , no orin，maybe you can use TensorRT8.6 ,and rebuild int8 model, version and hardware affect the optimization.Such a large amount of time should not be spent on lightweight backbones and fuse necks

LiuXuan66 commented 1 year ago

Thanks a lot. I will try to update the version of tensorrt.

ASONG0506 commented 1 year ago

the time consumption on orin platform is as below

==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.01830 ms
[⏰ Camera Backbone]:   8.91315 ms
[⏰ BEV vision transform]:      0.71488 ms
[⏰ BEV fuse head]:     66.66816 ms
[⏰ BEV postprocess]:   4.80109 ms
Total: 81.097 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.17738 ms
[⏰ Camera Backbone]:   8.15795 ms
[⏰ BEV vision transform]:      0.71149 ms
[⏰ BEV fuse head]:     66.38195 ms
[⏰ BEV postprocess]:   4.75507 ms
Total: 80.006 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.16928 ms
[⏰ Camera Backbone]:   8.14246 ms
[⏰ BEV vision transform]:      1.20810 ms
[⏰ BEV fuse head]:     66.38873 ms
[⏰ BEV postprocess]:   4.76374 ms
Total: 80.503 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.15174 ms
[⏰ Camera Backbone]:   8.14074 ms
[⏰ BEV vision transform]:      1.21091 ms
[⏰ BEV fuse head]:     66.38090 ms
[⏰ BEV postprocess]:   4.75018 ms
Total: 80.483 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.15520 ms
[⏰ Camera Backbone]:   8.14307 ms
[⏰ BEV vision transform]:      1.21088 ms
[⏰ BEV fuse head]:     66.37798 ms
[⏰ BEV postprocess]:   4.73478 ms
Total: 80.467 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.12976 ms
[⏰ Camera Backbone]:   8.14323 ms
[⏰ BEV vision transform]:      1.20963 ms
[⏰ BEV fuse head]:     66.39207 ms
[⏰ BEV postprocess]:   4.73328 ms
Total: 80.478 ms
=============================================

is there something wrong?

QiTianDaShengDaShi commented 1 year ago

我在orin上耗时：

`==================FastBEV=================== [⏰ [NoSt] ImageNrom]: 9.78643 ms [⏰ Camera Backbone]: 62.60304 ms [⏰ BEV vision transform]: 1.22509 ms [⏰ BEV fuse head]: 16.17450 ms [⏰ BEV postprocess]: 4.65139 ms Total: 84.654 ms

==================FastBEV=================== [⏰ [NoSt] ImageNrom]: 6.93651 ms [⏰ Camera Backbone]: 18.19347 ms [⏰ BEV vision transform]: 1.16883 ms [⏰ BEV fuse head]: 15.51933 ms [⏰ BEV postprocess]: 4.57638 ms Total: 39.458 ms

==================FastBEV=================== [⏰ [NoSt] ImageNrom]: 7.49053 ms [⏰ Camera Backbone]: 16.25187 ms [⏰ BEV vision transform]: 1.10752 ms [⏰ BEV fuse head]: 8.14979 ms [⏰ BEV postprocess]: 5.24106 ms Total: 30.750 ms

==================FastBEV=================== [⏰ [NoSt] ImageNrom]: 4.56509 ms [⏰ Camera Backbone]: 16.32819 ms [⏰ BEV vision transform]: 1.07542 ms [⏰ BEV fuse head]: 12.50774 ms [⏰ BEV postprocess]: 4.60144 ms Total: 34.513 ms

==================FastBEV=================== [⏰ [NoSt] ImageNrom]: 8.25453 ms [⏰ Camera Backbone]: 15.61411 ms [⏰ BEV vision transform]: 1.10198 ms [⏰ BEV fuse head]: 8.41549 ms [⏰ BEV postprocess]: 6.73754 ms Total: 31.869 ms

==================FastBEV=================== [⏰ [NoSt] ImageNrom]: 4.80746 ms [⏰ Camera Backbone]: 16.43366 ms [⏰ BEV vision transform]: 1.10966 ms [⏰ BEV fuse head]: 11.59578 ms [⏰ BEV postprocess]: 5.07693 ms Total: 34.216 ms

`

int8的。

GeLink9999 commented 1 year ago

the time consumption on orin platform is as below

==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.01830 ms
[⏰ Camera Backbone]:   8.91315 ms
[⏰ BEV vision transform]:      0.71488 ms
[⏰ BEV fuse head]:     66.66816 ms
[⏰ BEV postprocess]:   4.80109 ms
Total: 81.097 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.17738 ms
[⏰ Camera Backbone]:   8.15795 ms
[⏰ BEV vision transform]:      0.71149 ms
[⏰ BEV fuse head]:     66.38195 ms
[⏰ BEV postprocess]:   4.75507 ms
Total: 80.006 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.16928 ms
[⏰ Camera Backbone]:   8.14246 ms
[⏰ BEV vision transform]:      1.20810 ms
[⏰ BEV fuse head]:     66.38873 ms
[⏰ BEV postprocess]:   4.76374 ms
Total: 80.503 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.15174 ms
[⏰ Camera Backbone]:   8.14074 ms
[⏰ BEV vision transform]:      1.21091 ms
[⏰ BEV fuse head]:     66.38090 ms
[⏰ BEV postprocess]:   4.75018 ms
Total: 80.483 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.15520 ms
[⏰ Camera Backbone]:   8.14307 ms
[⏰ BEV vision transform]:      1.21088 ms
[⏰ BEV fuse head]:     66.37798 ms
[⏰ BEV postprocess]:   4.73478 ms
Total: 80.467 ms
=============================================
==================FastBEV===================
[⏰ [NoSt] ImageNrom]:  3.12976 ms
[⏰ Camera Backbone]:   8.14323 ms
[⏰ BEV vision transform]:      1.20963 ms
[⏰ BEV fuse head]:     66.39207 ms
[⏰ BEV postprocess]:   4.73328 ms
Total: 80.478 ms
=============================================

is there something wrong?

HI Have you solved this problem? Thanks. @ASONG0506 @QiTianDaShengDaShi @LiuXuan66