Dynamic CLASS_NUM for YoloLayerPlugin

[AdamChrist]

Hello, I encountered a problem. I used Triton Inference Server to deploy models with different class num. How can YoloLayerPlugin solve this problem?

[philipp-schmidt]

Hello. Have you tried to change anything in the code yourself already? If you have and it did not work I'm glad to have a look at the specific issue.

[AdamChrist]

Number of classes CLASS_NUM defined in yololayer.h . Changing the number can be applied to different models.But they can’t work at the same time.Because different models share the same plugin. Is there any way to dynamically change the CLASS_NUM in the plugin, perhaps based on the name of the model?

[philipp-schmidt]

Yes. You would have to name the layers differently. Each layer has a unique namespace string:

https://github.com/isarsoft/yolov4-triton-tensorrt/blob/31aa9f9efe5abea78d57fefd082f30e9fbda5da1/layers/yololayer.cu#L129-L132

This namespace string is resued when the network is build and also when it is deserialized:

https://github.com/isarsoft/yolov4-triton-tensorrt/blob/4edbe849f111324d58eda5f14fe1e4e9b098d4cd/networks/yolov4.h#L325-L329

You should be able to build one plugin for each model (with diffferent CLASS_NUM) that you have and name them differently. Then you need to load all plugins. Let me know if that works for you.

[AdamChrist]

@philipp-schmidt thank you for your reply. I deployed the model using Triton Inference Server, so there is no way to use the method you provided. Triton Inference Server can only use one libraries ,like this:

$ LD_PRELOAD=/plugins/liblayerplugin.so --model-repository=/tmp/models ...

Can I distinguish the name of the current inference model in the plugin of liblayerplugin.so? Or is there another way to know the class num of the current inference model ?

[philipp-schmidt]

you can load multiple .so files via LD_PRELOAD. Google LD_PRELOAD.

It's probably semicolon to separate them.

[philipp-schmidt]

Its space or colon. https://stackoverflow.com/questions/8474363/specifying-multiple-files-with-ld-preload

[philipp-schmidt]

You can also just have multiple layers with different names in a single plugin. Both is fine.

[AdamChrist]

@philipp-schmidt Thank you very much for your help, I will try it

[AdamChrist]

@philipp-schmidt Hello, I still have questions about this issue. For example, I have two models, the class num of model A is 2, and the class_num of model B is 8. I will deploy them to Triton Inference Server. How can I make Two models use different plugins liblayerplugin? Thank you

[michaelwithu]

have you solve the problem you proposed

[philipp-schmidt]

Duplicate the yolo layer class and give the new layer updated properties and also a different namespace (look above). Then create a copy of your network class and chose the new layer implementation instead of the old. You will end up with two engines (execute twice) and a single plugin with two layer implementations.

[michaelwithu]

would you mind explain why you using the liblayerplugin.so,can it takes the place of “config.pbtxt” ？

[philipp-schmidt]

No, liblayerplugin implements the yolo layer for TensorRT as it is not supported in TensorRT out of the box.

[michaelwithu]

No, liblayerplugin implements the yolo layer for TensorRT as it is not supported in TensorRT out of the box.

Can one liblayerplugin be used for n yolov4 models which are different in CLASS_NUM only. i dont understand how triton inference server to use this liblayerplugin. is it called "Backend Shared Library" in the official documentation???

[chull434]

@AdamChrist did you ever get triton to load more than one plugin?

[AdamChrist]

@chull434 I have implemented this feature with reference to tensorrtx ,you can also use yolov5, which has supported different models.