search

ceccocats / tkDNN

Deep neural network library and toolkit to do high performace inference on NVIDIA jetson platforms
GNU General Public License v2.0
718 stars 209 forks source link

Triton: Serialized engine contains plugin, but no plugin factory was provided. #247

Open GowthamKudupudi opened 3 years ago

GowthamKudupudi commented 3 years ago

I have generated a .rt (.plan) file of a trained yolo4tiny Darknet model by passing the weights, input_bins and out_bins to tests/darknet/yolo4tiny.cpp.

When I try to serve the generated .plan file with Triton server, it fails to load the model with error:

...
...
I0721 04:58:21.624484 27811 backend_factory.h:44] Create TritonBackendFactory
I0721 04:58:21.624551 27811 plan_backend_factory.cc:48] Create PlanBackendFactory
I0721 04:58:21.624582 27811 plan_backend_factory.cc:55] Registering TensorRT Plugins
I0721 04:58:21.624683 27811 logging.cc:52] Registered plugin creator - ::GridAn$hor_TRT version 1
I0721 04:58:21.624707 27811 logging.cc:52] Registered plugin creator - ::NMS_TRT version 1
I0721 04:58:21.624771 27811 logging.cc:52] Registered plugin creator - ::Reorg_TRT version 1
I0721 04:58:21.624801 27811 logging.cc:52] Registered plugin creator - ::Region_TRT version 1
I0721 04:58:21.624819 27811 logging.cc:52] Registered plugin creator - ::Clip_TRT version 1
I0721 04:58:21.624831 27811 logging.cc:52] Registered plugin creator - ::LReLU_TRT version 1
I0721 04:58:21.624868 27811 logging.cc:52] Registered plugin creator - ::PriorBox_TRT version 1
I0721 04:58:21.624903 27811 logging.cc:52] Registered plugin creator - ::Normalize_TRT version 1
I0721 04:58:21.624920 27811 logging.cc:52] Registered plugin creator - ::RPROI_TRT version 1
I0721 04:58:21.624952 27811 logging.cc:52] Registered plugin creator - ::Batche$NMS_TRT version 1
I0721 04:58:21.624971 27811 logging.cc:52] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
I0721 04:58:21.625014 27811 logging.cc:52] Registered plugin creator - ::FlattenConcat_TRT version 1
I0721 04:58:21.625035 27811 logging.cc:52] Registered plugin creator - ::CropAndResize version 1
I0721 04:58:21.625053 27811 logging.cc:52] Registered plugin creator - ::DetectionLayer_TRT version 1
I0721 04:58:21.625069 27811 logging.cc:52] Registered plugin creator - ::Proposal version 1
I0721 04:58:21.625084 27811 logging.cc:52] Registered plugin creator - ::ProposalLayer_TRT version 1
I0721 04:58:21.625108 27811 logging.cc:52] Registered plugin creator - ::PyramidROIAlign_TRT version 1
I0721 04:58:21.625133 27811 logging.cc:52] Registered plugin creator - ::ResizeNearest_TRT version 1
I0721 04:58:21.625156 27811 logging.cc:52] Registered plugin creator - ::Split version 1
I0721 04:58:21.625174 27811 logging.cc:52] Registered plugin creator - ::SpecialSlice_TRT version 1
I0721 04:58:21.625204 27811 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 1
E0721 04:58:23.444743 27811 logging.cc:43] deserializationUtils.cpp (635) - Serialization Error in load: 0 (Serialized engine contains plugin, but no plugin factory was provided. To deserialize an engine without a factory, please use IPluginV2 instead.)
E0721 04:58:23.446126 27811 logging.cc:43] INVALID_STATE: std::exception
E0721 04:58:23.447053 27811 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed.
...
...

When I try to run Triton with LD_PRELOAD=libkernels.so it throws the below additional errors

E0721 10:11:41.959468 27857 logging.cc:43] coreReadArchive.cpp (32) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
E0721 10:11:41.959607 27857 logging.cc:43] INVALID_STATE: std::exception
E0721 10:11:41.959627 27857 logging.cc:43] INVALID_CONFIG: Deserialize the cudaengine failed.
E0721 10:11:41.961273 27857 logging.cc:43] coreReadArchive.cpp (32) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
E0721 10:11:41.961326 27857 logging.cc:43] INVALID_STATE: std::exception
E0721 10:11:41.961346 27857 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed.

how to create custom plugins .so with tkDNN? is libkernels.so that was created while building the tkDNN, the custom plugins shared object? Can I generate the .rt (.plan) file by using inbuilt TRT plugins mentioned in the above log in place of custom plugins?

I have built and run the tritonserver on the same container in which tkDNN is built and .rt file is generated.

TheExDeus commented 2 years ago

The current tkdnn doesn't support plugins registry (which is why it doesn't work on trt8). You can try building the .so in this branch: https://github.com/TheExDeus/tkDNN/tree/feature/tensorrt8_support

mive93 commented 2 years ago

TensorRT8 is now supported on tensorrt8 branch.