search

isarsoft / yolov4-triton-tensorrt

This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server
http://www.isarsoft.com
Other
276 stars 63 forks source link

TensorRT 8 Support #55

Closed Michelvl92 closed 2 years ago

Michelvl92 commented 2 years ago

I have tried to build yolov4 with the tutorial, but instead of using Nvidia tensorrt docker version 21.03 version, I used 21.09 (with TensorRT 8.0.3) which resulted in the following errors:

liermlv@v2119:/data/liermlv/gitlab/external$ docker run --gpus all -it --rm -v /data/liermlv/^Cvcr.io/nvidia/tensorrt:21.09-py3
liermlv@v2119:/data/liermlv/gitlab/external$ tmux new -s tensorrt-conv
root@786e221fbf7e:/workspace# cd /external/yolov4-triton-tensorrt/                                                                                                        [66/66]
root@786e221fbf7e:/external/yolov4-triton-tensorrt# mkdir build
root@786e221fbf7e:/external/yolov4-triton-tensorrt# cd build
root@786e221fbf7e:/external/yolov4-triton-tensorrt/build# cmake ..
-- The CXX compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "11.4")
embed_platform off
-- Configuring done
-- Generating done
-- Build files have been written to: /external/yolov4-triton-tensorrt/build
root@786e221fbf7e:/external/yolov4-triton-tensorrt/build# make
[ 25%] Building NVCC (Device) object CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o
/external/yolov4-triton-tensorrt/layers/yololayer.h(59): error: member function declared with "override" does not override a base class member

/external/yolov4-triton-tensorrt/layers/yololayer.h(108): error: function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int
32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)"
/usr/include/x86_64-linux-gnu/NvInferRuntimeCommon.h(836): here is inaccessible

/external/yolov4-triton-tensorrt/layers/yololayer.h(59): warning: function "nvinfer1::IPluginV2::enqueue(int32_t, const void *const *, void *const *, void *, cudaStream_t)" is h
idden by "nvinfer1::YoloLayerPlugin::enqueue" -- virtual function override intended?

/external/yolov4-triton-tensorrt/layers/yololayer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int3
2_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::con
figurePlugin" -- virtual function override intended?

/external/yolov4-triton-tensorrt/layers/yololayer.h(91): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::detachFromContext" is incompatible with
that of overridden function "nvinfer1::IPluginV2Ext::detachFromContext"

/external/yolov4-triton-tensorrt/layers/yololayer.h(89): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::configurePlugin" is incompatible with th
at of overridden function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::PluginTensorDesc *, int32_t, const nvinfer1::PluginTensorDesc *, int32_t)"

/external/yolov4-triton-tensorrt/layers/yololayer.h(87): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::attachToContext" is incompatible with th
at of overridden function "nvinfer1::IPluginV2Ext::attachToContext"

/external/yolov4-triton-tensorrt/layers/yololayer.h(85): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::canBroadcastInputAcrossBatch" is incompa
tible with that of overridden function "nvinfer1::IPluginV2Ext::canBroadcastInputAcrossBatch"

/external/yolov4-triton-tensorrt/layers/yololayer.h(83): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::isOutputBroadcastAcrossBatch" is incompa
tible with that of overridden function "nvinfer1::IPluginV2Ext::isOutputBroadcastAcrossBatch"

/external/yolov4-triton-tensorrt/layers/yololayer.h(81): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDataType" is incompatible with
that of overridden function "nvinfer1::IPluginV2Ext::getOutputDataType"

/external/yolov4-triton-tensorrt/layers/yololayer.h(79): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginNamespace" is incompatible with
 that of overridden function "nvinfer1::IPluginV2::getPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(77): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::setPluginNamespace" is incompatible with
 that of overridden function "nvinfer1::IPluginV2::setPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of over
ridden function "nvinfer1::IPluginV2Ext::clone"

/external/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2::clone"

/external/yolov4-triton-tensorrt/layers/yololayer.h(73): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::destroy" is incompatible with that of overridden function "nvinfer1::IPluginV2::destroy"

/external/yolov4-triton-tensorrt/layers/yololayer.h(71): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginVersion"

/external/yolov4-triton-tensorrt/layers/yololayer.h(69): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginType" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginType

/external/yolov4-triton-tensorrt/layers/yololayer.h(65): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::supportsFormatCombination" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::supportsFormatCombination"

/external/yolov4-triton-tensorrt/layers/yololayer.h(63): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::serialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::serialize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(61): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getSerializationSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getSerializationSize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(57): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getWorkspaceSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getWorkspaceSize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(55): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::terminate" is incompatible with that of overridden function "nvinfer1::IPluginV2::terminate"

/external/yolov4-triton-tensorrt/layers/yololayer.h(53): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::initialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::initialize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(51): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDimensions" is incompatible wit
h that of overridden function "nvinfer1::IPluginV2::getOutputDimensions"

/external/yolov4-triton-tensorrt/layers/yololayer.h(46): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getNbOutputs" is incompatible with that
of overridden function "nvinfer1::IPluginV2::getNbOutputs"

/external/yolov4-triton-tensorrt/layers/yololayer.h(133): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginNamespace" is incompatible w
ith that of overridden function "nvinfer1::IPluginCreator::getPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(128): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::setPluginNamespace" is incompatible w
ith that of overridden function "nvinfer1::IPluginCreator::setPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(126): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::deserializePlugin" is incompatible wi
th that of overridden function "nvinfer1::IPluginCreator::deserializePlugin"

/external/yolov4-triton-tensorrt/layers/yololayer.h(124): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::createPlugin" is incompatible with th
at of overridden function "nvinfer1::IPluginCreator::createPlugin"

/external/yolov4-triton-tensorrt/layers/yololayer.h(122): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getFieldNames" is incompatible with t
hat of overridden function "nvinfer1::IPluginCreator::getFieldNames"

/external/yolov4-triton-tensorrt/layers/yololayer.h(120): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginVersion" is incompatible wit
h that of overridden function "nvinfer1::IPluginCreator::getPluginVersion"

/external/yolov4-triton-tensorrt/layers/yololayer.h(118): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginName" is incompatible with t
hat of overridden function "nvinfer1::IPluginCreator::getPluginName"

/external/yolov4-triton-tensorrt/layers/yololayer.cu(173): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
            pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/external/yolov4-triton-tensorrt/layers/yololayer.cu(412): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
            pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/external/yolov4-triton-tensorrt/layers/yololayer.cu(419): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
            pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

33 errors detected in the compilation of "/external/yolov4-triton-tensorrt/layers/yololayer.cu".
CMake Error at layerplugin_generated_yololayer.cu.o.cmake:279 (message):
  Error generating file
  /external/yolov4-triton-tensorrt/build/CMakeFiles/layerplugin.dir/layers/./layerplugin_generated_yololayer.cu.o

make[2]: *** [CMakeFiles/layerplugin.dir/build.make:65: CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:109: CMakeFiles/layerplugin.dir/all] Error 2
make: *** [Makefile:84: all] Error 2
philipp-schmidt commented 2 years ago

TensorRT 8 changed the way plugins work. I did not have time to look into it yet. Contributions are very welcomed!

philipp-schmidt commented 2 years ago

Usually the computational part of the plugin stays the same and only the interface around it changes, so it should not be too big of a change.

revanb88 commented 2 years ago

I am also facing same issue, could you please help me. I used tensorrt:21.08 got this error

/yolov4-triton-tensorrt/layers/yololayer.h(59): error: member function declared with "override" does not override a base class member

/yolov4-triton-tensorrt/layers/yololayer.h(108): error: function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::Dims , int32_t, const nvinfer1::Dims , int32_t, const nvinfer1::DataType , const nvinfer1::DataType , const __nv_bool , const __nv_bool , nvinfer1::PluginFormat, int32_t)" /usr/include/x86_64-linux-gnu/NvInferRuntimeCommon.h(836): here is inaccessible

/yolov4-triton-tensorrt/layers/yololayer.h(59): warning: function "nvinfer1::IPluginV2::enqueue(int32_t, const void const , void const , void *, cudaStream_t)" is hidden by "nvinfer1::YoloLayerPlugin::enqueue" -- virtual function override intended?

/yolov4-triton-tensorrt/layers/yololayer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims , int32_t, const nvinfer1::Dims , int32_t, const nvinfer1::DataType , const nvinfer1::DataType , const __nv_bool , const __nv_bool , nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::configurePlugin" -- virtual function override intended?

/yolov4-triton-tensorrt/layers/yololayer.h(91): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::detachFromContext" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::detachFromContext"

/yolov4-triton-tensorrt/layers/yololayer.h(89): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::configurePlugin" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::PluginTensorDesc , int32_t, const nvinfer1::PluginTensorDesc , int32_t)"

/yolov4-triton-tensorrt/layers/yololayer.h(87): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::attachToContext" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::attachToContext"

/yolov4-triton-tensorrt/layers/yololayer.h(85): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::canBroadcastInputAcrossBatch" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::canBroadcastInputAcrossBatch"

/yolov4-triton-tensorrt/layers/yololayer.h(83): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::isOutputBroadcastAcrossBatch" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::isOutputBroadcastAcrossBatch"

/yolov4-triton-tensorrt/layers/yololayer.h(81): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDataType" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::getOutputDataType"

/yolov4-triton-tensorrt/layers/yololayer.h(79): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(77): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::setPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginV2::setPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::clone"

/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2::clone"

/yolov4-triton-tensorrt/layers/yololayer.h(73): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::destroy" is incompatible with that of overridden function "nvinfer1::IPluginV2::destroy"

/yolov4-triton-tensorrt/layers/yololayer.h(71): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginVersion"

/yolov4-triton-tensorrt/layers/yololayer.h(69): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginType" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginType"

/yolov4-triton-tensorrt/layers/yololayer.h(65): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::supportsFormatCombination" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::supportsFormatCombination"

/yolov4-triton-tensorrt/layers/yololayer.h(63): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::serialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::serialize"

/yolov4-triton-tensorrt/layers/yololayer.h(61): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getSerializationSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getSerializationSize"

/yolov4-triton-tensorrt/layers/yololayer.h(57): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getWorkspaceSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getWorkspaceSize"

/yolov4-triton-tensorrt/layers/yololayer.h(55): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::terminate" is incompatible with that of overridden function "nvinfer1::IPluginV2::terminate"

/yolov4-triton-tensorrt/layers/yololayer.h(53): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::initialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::initialize"

/yolov4-triton-tensorrt/layers/yololayer.h(51): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDimensions" is incompatible with that of overridden function "nvinfer1::IPluginV2::getOutputDimensions"

/yolov4-triton-tensorrt/layers/yololayer.h(46): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getNbOutputs" is incompatible with that of overridden function "nvinfer1::IPluginV2::getNbOutputs"

/yolov4-triton-tensorrt/layers/yololayer.h(133): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(128): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::setPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginCreator::setPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(126): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::deserializePlugin" is incompatible with that of overridden function "nvinfer1::IPluginCreator::deserializePlugin"

/yolov4-triton-tensorrt/layers/yololayer.h(124): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::createPlugin" is incompatible with that of overridden function "nvinfer1::IPluginCreator::createPlugin"

/yolov4-triton-tensorrt/layers/yololayer.h(122): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getFieldNames" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getFieldNames"

/yolov4-triton-tensorrt/layers/yololayer.h(120): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginVersion"

/yolov4-triton-tensorrt/layers/yololayer.h(118): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginName" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginName"

/yolov4-triton-tensorrt/layers/yololayer.cu(173): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed: pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/yolov4-triton-tensorrt/layers/yololayer.cu(412): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed: pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/yolov4-triton-tensorrt/layers/yololayer.cu(419): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed: pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

33 errors detected in the compilation of "/yolov4-triton-tensorrt/layers/yololayer.cu". CMake Error at layerplugin_generated_yololayer.cu.o.cmake:279 (message): Error generating file /yolov4-triton-tensorrt/build/CMakeFiles/layerplugin.dir/layers/./layerplugin_generated_yololayer.cu.o

make[2]: [CMakeFiles/layerplugin.dir/build.make:65: CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o] Error 1 make[1]: [CMakeFiles/Makefile2:109: CMakeFiles/layerplugin.dir/all] Error 2 make: *** [Makefile:84: all] Error 2

chull434 commented 2 years ago

yeah ran into the same issue when I update to jetson jetpack 4.6 from 4.5 and tensorRT upgraded to 8.0.1

chull434 commented 2 years ago

I followed this pr's changes and got it to at least build now https://github.com/wang-xinyu/tensorrtx/pull/540

Also note need to update the docker image to a TRT8 one, such as nvcr.io/nvidia/tensorrt:21.09-py3

revanb88 commented 2 years ago

Thanks a lot. I will go through it.

On Wed, Nov 3, 2021 at 9:00 PM chull434 @.***> wrote:

I followed this pr's changes and got it to at least build now wang-xinyu/tensorrtx#540 https://github.com/wang-xinyu/tensorrtx/pull/540

Also note need to update the docker image to a TRT8 one, such as nvcr.io/nvidia/tensorrt:21.09-py3

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/isarsoft/yolov4-triton-tensorrt/issues/55#issuecomment-959453221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWDAXTIJB4DDQHIBALD76KLUKFIRHANCNFSM5F5DGZ3Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks and Regards Dr. Revanasiddappa M Biradar Data Scientist Bengaluru

ohlr commented 2 years ago

@revanb88, @chull434 - would you like to make this a PR?

ohlr commented 2 years ago

@Michelvl92 did you find a way to support TRT8?

jenfredwell commented 2 years ago

I've also ended up here with the same errors trying to use the 21.08 version. Curious about the status?

philipp-schmidt commented 2 years ago

No support yet. We will update this to TensorRT 8 at some point, no roadmap though. PR from someone else will probably be faster. Should only be a matter of applying necessary changes from the patch notes from nvidia to the new plugin interface.

philipp-schmidt commented 2 years ago

Newest v1.4.0 release comes with support for TRT 8. Tested with NGC TensorRT 21.10 and Triton 21.10 with 470 driver. Performance gain compared to previous 1.3.0 release is at about 19.5%.

Did not have the time to run and check the python client, so feel free to report any issues with that.