search

xuanandsix / GFPGAN-onnxruntime-demo

This is the onnxruntime inference code for GFP-GAN: Towards Real-World Blind Face Restoration with Generative Facial Prior (CVPR 2021). Official code: https://github.com/TencentARC/GFPGAN
118 stars 16 forks source link

Any chance we can load the onnx model and run it into TensorRT #2

Open nelsontseng0704 opened 1 year ago

nelsontseng0704 commented 1 year ago

Such a great work. I am looking for quicker inference for GFPGAN. Is there any chance we can load the GFPGAN.onnx and run it on TensorRT? Looking forward to your suggestion.

Thanks.

lschaupp commented 10 months ago

Yes I made it work. I am getting 70 ms runtime on 3060 RTX for the 512 upscale model. (GFPGAN 1.4) using TensorRT. (without pre and post processing) However the ONNX model itself has some quality decrease in comparison to the original model for some reason.

nelsontseng0704 commented 10 months ago

Thanks for the sharing. I follow your fork from Face-Restoration-TensorRT and I tried to build the same fgpgan tensorrt, however I encounter the following error, do I miss something on the CMakeLists.txt?

[ 20%] Building CXX object CMakeFiles/demo.dir/demo.cpp.o In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:26, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeBase.h: In member function ‘virtual bool nvinfer1::IGpuAllocator::deallocate(void*)’: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeBase.h:479:26: warning: ‘virtual void nvinfer1::IGpuAllocator::free(void*)’ is deprecated [-Wdeprecated-declarations] 479 | this->free(memory); | ^ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeBase.h:410:33: note: declared here 410 | TRT_DEPRECATED virtual void free(void* const memory) noexcept = 0; | ^~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h: At global scope: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:866:104: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 866 | virtual IPluginV2* createPlugin(AsciiChar const* name, PluginFieldCollection const* fc) noexcept = 0; | ^ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:878:11: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 878 | = 0; | ^ /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntime.h:22, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:17, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferImpl.h:648:47: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 648 | virtual IPluginV2& getPlugin() noexcept = 0; | ^ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntime.h:22, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:17, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferImpl.h:1014:113: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 1014 | virtual IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) noexcept = 0; | ^ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:3647:28: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 3647 | IPluginV2& getPlugin() noexcept | ^~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:6965:88: warning: ‘IFullyConnectedLayer’ is deprecated [-Wdeprecated-declarations] 6965 | ITensor& input, int32_t nbOutputs, Weights kernelWeights, Weights biasWeights) noexcept | ^~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:1518:22: note: declared here 1518 | class TRT_DEPRECATED IFullyConnectedLayer : public ILayer | ^~~~~~~~~~~~~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:7542:101: warning: ‘IRNNv2Layer’ is deprecated [-Wdeprecated-declarations] 7542 | ITensor& input, int32_t layerCount, int32_t hiddenSize, int32_t maxSeqLen, RNNOperation op) noexcept | ^~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:3349:22: note: declared here 3349 | class TRT_DEPRECATED IRNNv2Layer : public ILayer | ^~~~~~~~~~~ In file included from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:7617:94: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations] 7617 | IPluginV2Layer* addPluginV2(ITensor* const* inputs, int32_t nbInputs, IPluginV2& plugin) noexcept | ^~~~~~~~ In file included from /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimeCommon.h:27, from /home/paperspace/TensorRT-8.6.1.6/include/NvInferLegacyDims.h:16, from /home/paperspace/TensorRT-8.6.1.6/include/NvInfer.h:16, from /home/paperspace/Face-Restoration-TensorRT/face_restoration.hpp:3, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:5: /home/paperspace/TensorRT-8.6.1.6/include/NvInferRuntimePlugin.h:97:22: note: declared here 97 | class TRT_DEPRECATED IPluginV2 | ^~~~~~~~~ In file included from /usr/include/c++/9/ratio:38, from /usr/include/c++/9/chrono:38, from /home/paperspace/Face-Restoration-TensorRT/demo.cpp:1: /usr/include/c++/9/type_traits: In instantiation of ‘struct std::is_polymorphic<FaceRestore>’: /home/paperspace/.local/lib/python3.9/site-packages/pybind11/include/pybind11/pybind11.h:1516:30: required from ‘class pybind11::class_<FaceRestore>’ /home/paperspace/Face-Restoration-TensorRT/demo.cpp:24:45: required from here /usr/include/c++/9/type_traits:709:12: error: invalid use of incomplete type ‘class FaceRestore’ 709 | struct is_polymorphic | ^~~~~~~~~~~~~~ compilation terminated due to -Wfatal-errors. make[2]: *** [CMakeFiles/demo.dir/build.make:76: CMakeFiles/demo.dir/demo.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/demo.dir/all] Error 2 make: *** [Makefile:91: all] Error 2

lschaupp commented 10 months ago

@nelsontseng0704 the fork has a different purpose. It is intended to construct a python wrapper around the C++ code for inference. For GFPGAN I made it work with the official repository.

Also it is heavily WIP. So no wonder you are getting some errors when trying to build it :)

JieLuChen commented 8 months ago

Yes I made it work. I am getting 70 ms runtime on 3060 RTX for the 512 upscale model. (GFPGAN 1.4) using TensorRT. (without pre and post processing) However the ONNX model itself has some quality decrease in comparison to the original model for some reason.

Got it working with model.onnx (including inference) also successfully converted model512.onnx to model2.engine But when running ./demo ../models/model2.engine -i ../images/test.png I always get Cuda failure: 700 Aborted (core dumped)

Do you know what the fix is for this?

lschaupp commented 6 months ago

You would need to debug the error within the C++ code. Cuda 700 is a standard error which does not tell much.