[RFE] Samples on custom plugins for ONNX models

[xiaoxiaotao]

I want to add a customer layer for C++ API ,but I can`t find demo.

[rajeevsrao]

@xiaoxiaotao

Check out this sample: https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/samplePlugin It contains a dummy plugin (custom layer) implementation for GEMM/FC for illustrative purposes: https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/samplePlugin/fcPlugin.h

[xiaoxiaotao]

@rajeevsrao It is a customer layer demo for caffe,but I use onnx model. I want to find a demo about ONNX,because I use pytorch .thank you .

[nfeng0105]

@xiaoxiaotao There are many onnx plugins under $TRT_SOURCE/parsers/onnx/, like Split.hpp, ResizeNearest.hpp and etc. They will be registered into system through 'REGISTER_TENSORRT_PLUGIN' automatically, so that your onnx model can be parsed directly during runtime.

[xiaoxiaotao]

@rajeevsrao @nfeng0105 Thank you very much

[ryo-takahashi-1206]

I hope someone would answer my question if he or she knows the latest onnx-tensorrt...

Background

I am thinking of speeding up a PyTorch function, F.interpolate(), by nvinfer1::IPluginV2.
The corresponding ONNX operator, Upsample, already has implementation.
However, because my settings of F.interpolate() are very limited, I expect my nvinfer1::IPluginV2 can shorten processing time by implementing it specialized for my settings.

Question

Thus, I'm currently wondering if my nvinfer1::IPluginV2 overrides the existing implementation
when my nvinfer1::IPluginCreator::getPluginName() returns "Upsample".
Please let me know if I'm confused about something ...

[OronG13]

Hello, I'm trying to register a pluginV2 to the Onnx parser as opposed above by @nfeng0105. I add the entire process description and some implemetation details here: https://devtalk.nvidia.com/default/topic/1068292/tensorrt/custom-layer-plugin-tensorrtc-nvuffparser-iuffparser-vs-tensorrt-c-nvonnxparser-iparser/

It isn't work with Onnx but only with Uff despite the fact that I implemented the pluginV2 excatly the same.

I think that maybe the root cause is the pluginV2 name which isn't exactly as it detailed in my Onnx model.

Please, Can someone share a full example of the pluginV2 itsel but also the higher level implementation that activate its methods. I just want to compare with my full implementations to be sure that I did it OK.

Additionally, what is the right way to verify what is the accurate layer name in the model in order to give my pluginV2 the right name?

Regard,

[rmccorm4]

Hey @OronG13,

I think the easiest way at the moment might be to (1) implement your custom plugin as a new importer in this file like here: https://github.com/onnx/onnx-tensorrt/blob/84b5be1d6fc03564f2c0dba85a2ee75bad242c2e/builtin_op_importers.cpp#L933-L943

(2) then re-build the ONNX parser per the README of this repo: https://github.com/NVIDIA/TensorRT#building-the-tensorrt-oss-components

(3) Re-run your application code that uses the ONNX parser

---

Can you share your segnet.onnx model? I'm curious to see if there's an easier way to add the plugin nodes.

[leimao]

So it sounds adding a plugin for ONNX models via ONNX-TRT path is much more complicated than adding a plugin for TensorFlow models via UFF-TRT path. @rmccorm4

[OronG13]

Thanks @rmccorm4, Your recommended technique to add a custom layer to the Onnx is very interestingת I didn't think about this way before.

I started to overview some operators importing logic and it seems:

• Much more complicated than the plugInV2 interface
• Inconsistent from one operator to others
• Demands a much deep understanding about the TensorRT mechanism and logic's flow

Is there any tutorial or guidance with examples how to add new operator?

I can tell you that I'm very frustrated because of this issue.... On the one hand, there is a lot of NVIDIA information about adding a plugin to the Onnx, Uff & Caffe parsers but on the other hand, they provide a full examples (PlugIn & higher level) only for the Uff & Caffe parsers and in the same time all TRT developer guides implicitly say that plugin is supported by the Onnx parser and refer to their plugins examples but no full examples are provided which can tell how to register it as they provide for the Uff & Caffe.

Regarding to my segnet.onnx model: Sure I will be glad to share it, any help and further investigation from your point of view will be much appreciated, but first I shall get my company approval to do that.

I will update soon...

Best regard,

[fanbeatsman]

I have the same problem. NVIDIA seems to be slowing the development of all parsers except ONNX, yet the ONNX path has the least amount of samples. Like you said, they don't even have one for working with Custom Plugins. I also have a custom plugin that's been working with UFF, but when switching to ONNX and putting the exact same name as the one used to REGISTER_TENSORRT_PLUGIN(), I get a No importer registered for op: DenseWarpLayer_TRT.

[OronG13]

Hello @rmccorm4 ,

As you asked, I want to share the segnet.onnx file but it too much big. Is there another way to share big files?

In the mean time I can provide a background information about the segnet model:

1. I downloaded it from this link: https://github.com/meetshah1995/pytorch-semseg
2. pytorch-semseg-master-segnetMaterial.zip Attached is a zip file which contsins: segnet.py segnetModelPrint.txt segnetModelTorchOnnxExportPrint.txt segNet.png - The Onnx model displayed with Netron application segnetModelTensorRTOnnxParserReport.txt - You can see that there is an exception raised due to partial support of the MaxPool layer. You can see in the following link what is the root cause of the exception and a formal approval from NVIDIA that they are not fully support this operation: https://devtalk.nvidia.com/default/topic/1068841/tensorrt/tensorrt-onnx-parser-parse-a-maxpool-layer-with-two-outputs-raise-en-exception-of-std-out_of_range/

Now, the only way that left is to implement a PlugIn for the MaxPool layer. NVIDIA planning to provide end2end example how to add a Plugin to the TRT Onnx parser, but till then, If you will be able to find the way and share it, it will be much appreciated.

A very interesting point is that inside segnetModelTorchOnnxExportPrint.txt report, you can see that the parser report about several PlugIns that were successfully registered, for example: _VERBOSE: Plugin creator registration succeeded - ONNXTRT_NAMESPACE::GridAnchorTRT

They are probably NVIDIA Onnx PlugIns...

Thanks,

[TrojanXu]

@xiaoxiaotao Just like what @rmccorm4 mentioned, you possibly have to hack into builtin_op_importers.cpp implementation of onnx-tensorrt to let the parser understand how to translate your onnx-op into trt-op with plugins support. I recently wrote a non-official sample here. Hope it helps for now. For official support, you perhaps have to wait.

[rmccorm4]

Hi.

There should be better support for using custom plugins with ONNX models in the next release. Until then, yes I believe the most feasible way is to add your custom op to the onnx-tensorrt source: https://github.com/onnx/onnx-tensorrt/blob/master/builtin_op_importers.cpp

I would expect that after editing the onnx-tensorrt source and re-building it, you should just be able to parse the model as usual. No plugin API code.

I have done something similar for modifying an existing op, but I haven’t done this myself for a new op yet. I don’t really know the limitations of this approach yet.

[OronG13]

Hello @eesweng, I didn't find a solution yet.

I agree with you that there are two options to solve this issue as described above, but I don't agree that option number#1 is the right one.

I'm asking a very simple question: Why for Uff (Tensorflow) the plugin interface is working perfect and for the Onnx (All frameworks) it isn't?

As I understood from the NVIDIA support team, till latest version of the TensorRT (7) the Onnx module isn't mature as the Uff module and that's why some capabilities are working for the Uff and not for the Onnx. According to their words, the next release will include a significant improvement regarding to their Onnx support including an additional operations and plugin interface and shall be published very soon (The original due date was this month). Additionally, they said that the Uff module will be deprecated in the next few releases. So, I agree with @rmccorm4 words about the next TensorRT release.

For me, it is not so urgent to solve this issue now so I prefer to wait for next TensorRT release.

Good luck!

[legolas123]

This is what you can do: Clone Tensorrt source code and write your plugin inside the "plugin" folder. You can look at the implementation of any other plugin present in the folder. There is a lot of boiler plate code required and it is better to follow that line by line. I will suggest you to implement a plugin that derives IPluginV2Ext because that works pretty well with tensorrt 7. Then you need to call it in the file InferPlugin.cpp. It will look something like initializePlugin(logger, libNamespace); The above thing takes care of the plugin implementation from tensorrt side. Now you need to tell tensorrt onnx interface about how to replace the symbolic op present in onnx with your implementation. So in the file parsers/onnx/builtin_op_importers.cpp, look at the DEFINE_BUILTIN_OP_IMPORTER boiler plate code for other plugins. This is how it must look for your plugin(sort of)

DEFINE_BUILTIN_OP_IMPORTER(YourPlugin) {
    nvinfer1::ITensor* tensor_ptr_inp = &convertToTensor(inputs.at(0), ctx);
    std::vector<nvinfer1::ITensor *> plugin_input = {tensor_ptr_inp};
    nvinfer1::IPluginV2* plugin = importPluginFromRegistry(ctx, pluginName, pluginVersion, node.name(), f);
    RETURN_FIRST_OUTPUT(ctx->network()->addPluginV2(plugin_input.data(), plugin_input.size() , *plugin));
}

After compiling the code, now you can run: ./build/parsers/onnx/onnx2trt input_onnx.onnx -o output_engine.engine

[rajeevsrao]

Please review our blogpost on deploying ONNX models with custom (plugin) layers using TensorRT 7.x+ and ONNX-GraphSurgeon: https://developer.nvidia.com/blog/estimating-depth-beyond-2d-using-custom-layers-on-tensorrt-and-onnx-models/

We will be adding more samples in the OSS repository in coming release. Thanks.

[zerollzeng]

I write a minimal onnx plugin template that might be helpful. see onnx plugin template

[OronG13]

Hello @zerollzeng, Thank you very much for sharing this information, I found it very useful. Regard,

[zerollzeng]

@OronG13 I'm glad it's helpful to you, if you have any question or feedback, welcome to bring me an issue :)

[Darshcg]

Thank you @zerollzeng. It is very useful.