search

jkjung-avt / tensorrt_demos

TensorRT MODNet, YOLOv4, YOLOv3, SSD, MTCNN, and GoogLeNet
https://jkjung-avt.github.io/
MIT License
1.74k stars 545 forks source link

onnx to trt failed.. "cannot get YoloLayer_TRT plugin creator" #476

Closed seungtaek94 closed 2 years ago

seungtaek94 commented 2 years ago

Hi. I try to convert yolo to trt.

I succeed to convert .cfg -> .onnx but failed .onnx -> .trt with Error "cannot get YoloLayer_TRT plugin creator".

What should I check now and what does mean that error message.?

I already make in plugins dir and get libyolo_layer.so lib.

jkjung-avt commented 2 years ago

Please provide the complete log of "onnx_to_tensorrt.py".

seungtaek94 commented 2 years ago

@jkjung-avt 

(trt) seungtaek@seungtaek:~/trt/tensorrt_demos/yolo$ python onnx_to_tensorrt.py -m yolov4-p5-640x384
Loading the ONNX file...
Adding yolo_layer plugins.
Traceback (most recent call last):
  File "onnx_to_tensorrt.py", line 212, in <module>
    main()
  File "onnx_to_tensorrt.py", line 201, in main
    args.model, args.int8, args.dla_core, args.verbose)
  File "onnx_to_tensorrt.py", line 123, in build_engine
    network = add_yolo_plugins(network, model_name, TRT_LOGGER)
  File "/home/seungtaek/trt/tensorrt_demos/yolo/plugins.py", line 123, in add_yolo_plugins
    raise RuntimeError('cannot get YoloLayer_TRT plugin creator')
RuntimeError: cannot get YoloLayer_TRT plugin creator

Here :)

jkjung-avt commented 2 years ago

You are not using the latest code... Please git pull the latest code, rebuild the "yolo_layer" plugin (make clean; make), and re-run "yolo_to_onnx.py" and "onnx_to_tensorrt.py".

The raise RuntimeError(...) is on line 122 (not 123) in the latest code.

https://github.com/jkjung-avt/tensorrt_demos/blob/1f8773b3f6091d2e3441c0d6ce5714e332421f8d/yolo/plugins.py#L121-L122

seungtaek94 commented 2 years ago

@jkjung-avt 

def get_plugin_creator(plugin_name, logger):
    """Get the TensorRT plugin creator."""
    trt.init_libnvinfer_plugins(logger, '')
    plugin_creator_list = trt.get_plugin_registry().plugin_creator_list
    for c in plugin_creator_list:
        #print(c.name)   # << here
        if c.name == plugin_name:
            return c
    return None

No, Im using latest version. I added only one line that print(c.name) in plugins.py. So that is why line number is different with latest code.

jkjung-avt commented 2 years ago

If this ctypes.cdll.LoadLibrary('../plugins/libyolo_layer.so') succeeds, "YoloLayer_TRT" plugin creator should have been registered into the TensorRT...

https://github.com/jkjung-avt/tensorrt_demos/blob/1f8773b3f6091d2e3441c0d6ce5714e332421f8d/yolo/plugins.py#L16-L17

Do you see something similar when you do print(c.name) on all TensorRT plugin creators?

Otherwise, which version of TensorRT are you using?

seungtaek94 commented 2 years ago

@jkjung-avt

I use TensorRT-7.1.3.4 with cuda 11.1.

And below is print(c.name) output.. YoloLayer_TRT is not registered.. But i dont know why.

CustomQKVToContextPluginDynamic
CustomEmbLayerNormPluginDynamic
RnRes2Br1Br2c_TRT
RnRes2Br1Br2c_TRT
CustomSkipLayerNormPluginDynamic
CustomSkipLayerNormPluginDynamic
CustomSkipLayerNormPluginDynamic
GroupNormalizationPlugin
CustomGeluPluginDynamic
CustomEmbLayerNormPluginDynamic
CgPersistentLSTMPlugin_TRT
CustomQKVToContextPluginDynamic
CustomQKVToContextPluginDynamic
CustomFCPluginDynamic
SingleStepLSTMPlugin
RnRes2Br2bBr2c_TRT
RnRes2Br2bBr2c_TRT
RnRes2FullFusion_TRT
SmallTileGEMM_TRT
RNNTEncoderPlugin
GridAnchor_TRT
NMS_TRT
Reorg_TRT
Region_TRT
Clip_TRT
LReLU_TRT
PriorBox_TRT
Normalize_TRT
RPROI_TRT
BatchedNMS_TRT
BatchedNMSDynamic_TRT
FlattenConcat_TRT
CropAndResize
DetectionLayer_TRT
Proposal
ProposalLayer_TRT
PyramidROIAlign_TRT
ResizeNearest_TRT
Split
SpecialSlice_TRT
InstanceNormalization_TRT
jkjung-avt commented 2 years ago

Beats me...

Did you see any errors/warnings when building the plugin?

seungtaek94 commented 2 years ago

@jkjung-avt

Below is all outputs when i do make in plugins forlder.

computes: 61
NVCCFLAGS: -gencode arch=compute_61,code=[sm_61,compute_61]
nvcc -ccbin g++ -I"/usr/local/cuda-11.1/include" -I"/usr/local/TensorRT-7.1.3.4/include" -I"/usr/local/include" -I"plugin" -gencode arch=compute_61,code=[sm_61,compute_61] -Xcompiler -fPIC -c -o yolo_layer.o yolo_layer.cu
yolo_layer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::configurePlugin" -- virtual function override intended?

yolo_layer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const bool *, const bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::configurePlugin" -- virtual function override intended?

g++ -shared -o libyolo_layer.so yolo_layer.o -L"/usr/local/cuda-11.1/lib64" -L"/usr/local/TensorRT-7.1.3.4/lib" -L"/usr/local/lib" -Wl,--start-group -lnvinfer -lnvparsers -lnvinfer_plugin -lcudnn -lcublas -lnvToolsExt -lcudart -lrt -ldl -lpthread -Wl,--end-group
jkjung-avt commented 2 years ago

Below is all outputs when i do make in plugins folder.

The log looks completely normal. (The virtual function override warnings could be safely ignored.) I don't really have on idea why you met this problem ("cannot get YoloLayer_TRT plugin creator")...

jkjung-avt commented 2 years ago

I'm closing this issue since I myself am not able to reproduce the problem.

sebawsky commented 2 years ago

Hello there,

I had exactly the same error, I could not get the script "onnx_to_tensorrt.py -m yolov4-416" to work. Has anyone found the solution?

jkjung-avt commented 2 years ago

@sebawsky Did you run "onnx_to_tensorrt.py" in the "yolo" subdirectory?

Refer to the relevant source code below.

https://github.com/jkjung-avt/tensorrt_demos/blob/1f8773b3f6091d2e3441c0d6ce5714e332421f8d/yolo/plugins.py#L16-L21

Are you able to see the "libyolo_layer.so" file if you do ls -l ../plugins/libyolo_layer.so?

sebawsky commented 2 years ago

@jkjung-avt Yes, I ran as it appears in the tutorial. I followed the tutorial on my Jetson Nano and I was able to make the network work without problems but in ubuntu18 I could not (I still have this error).

Greetings

jkjung-avt commented 2 years ago

@sebawsky Make sure your TensorRT libraries (libnvinfer.so, etc.) are in your shared library paths.

Reference: https://unix.stackexchange.com/questions/22926/where-do-executables-look-for-shared-objects-at-runtime

akashAD98 commented 2 years ago

Im also getting the same issue

i have installed TensorVersion: 8.0.1.6 on tesla v4 (ubuntu 20.04 ) amazon machine

File "onnx_to_tensorrt.py", line 212, in main() File "onnx_to_tensorrt.py", line 200, in main engine = build_engine( File "onnx_to_tensorrt.py", line 123, in build_engine network = add_yolo_plugins(network, model_name, TRT_LOGGER) File "/home/ubuntu/tensorrt_demos/yolo/plugins.py", line 122, in add_yolo_plugins raise RuntimeError('cannot get YoloLayer_TRT plugin creator')

cannot get YoloLayer_TRT plugin creator

i have sucesfully converte3d yolo yo onnx , but getting the exactly same issue, Even i checked path of libyolo_layer.so & added here still getting the same issue

try: ctypes.cdll.LoadLibrary('/home/ubuntu/tensorrt_demos/plugins/libyolo_layer.so')

Your help is really appreciated.Thanks @jkjung-avt

i have attachd all screenshots ,please have look cuda 11.3 tensorrt 8.0.1.6 ubantu 20.04 tesla v4

image

image image image

also i have printed this C_NAMES_ CustomQKVToContextPluginDynamic C_NAMES___ CustomEmbLayerNormPluginDynamic C_NAMES___ RnRes2Br1Br2c_TRT CNAMES RnRes2Br1Br2c_TRT C_NAMES_ CustomSkipLayerNormPluginDynamic C_NAMES___ CustomSkipLayerNormPluginDynamic C_NAMES___ CustomSkipLayerNormPluginDynamic C_NAMES___ GroupNormalizationPlugin C_NAMES___ CustomGeluPluginDynamic C_NAMES___ CustomEmbLayerNormPluginDynamic C_NAMES___ CgPersistentLSTMPlugin_TRT CNAMES CustomQKVToContextPluginDynamic C_NAMES_ CustomQKVToContextPluginDynamic C_NAMES___ CustomFCPluginDynamic C_NAMES___ SingleStepLSTMPlugin C_NAMES___ RnRes2Br2bBr2c_TRT CNAMES RnRes2Br2bBr2c_TRT C_NAMES_ RnRes2FullFusion_TRT CNAMES SmallTileGEMM_TRT C_NAMES_ RNNTEncoderPlugin C_NAMES___ GridAnchor_TRT CNAMES NMS_TRT C_NAMES_ Reorg_TRT CNAMES Region_TRT C_NAMES_ Clip_TRT CNAMES LReLU_TRT C_NAMES_ PriorBox_TRT CNAMES Normalize_TRT C_NAMES_ RPROI_TRT CNAMES BatchedNMS_TRT C_NAMES_ BatchedNMSDynamic_TRT CNAMES FlattenConcat_TRT C_NAMES_ CropAndResize C_NAMES___ DetectionLayer_TRT CNAMES Proposal C_NAMES_ ProposalLayer_TRT CNAMES PyramidROIAlign_TRT C_NAMES_ ResizeNearest_TRT CNAMES Split C_NAMES_ SpecialSlice_TRT CNAMES InstanceNormalization_TRT

jkjung-avt commented 2 years ago

Please check the following (an example when I run it on my Jetson Xavier NX, JetPack-4.5). Make sure "ldd" is able to find all libraries which "libyolo_layer.so" is dependent upon.

$ cd yolo/
$ ldd ../plugins/libyolo_layer.so
        linux-vdso.so.1 (0x0000007fac0a2000)
        libnvinfer.so.7 => /usr/lib/aarch64-linux-gnu/libnvinfer.so.7 (0x0000007f9ebcc000)
        libcudart.so.10.2 => /usr/local/cuda/lib64/libcudart.so.10.2 (0x0000007f9eb58000)
        libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f9e9c4000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f9e9a0000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f9e847000)
        /lib/ld-linux-aarch64.so.1 (0x0000007fac076000)
        libcublas.so.10 => /usr/lib/aarch64-linux-gnu/libcublas.so.10 (0x0000007f99b69000)
        libcudnn.so.8 => /usr/lib/aarch64-linux-gnu/libcudnn.so.8 (0x0000007f99b2b000)
        libmyelin.so.1 => /usr/lib/aarch64-linux-gnu/libmyelin.so.1 (0x0000007f9969d000)
        libnvrtc.so.10.2 => /usr/local/cuda/lib64/libnvrtc.so.10.2 (0x0000007f9826c000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f98257000)
        libnvdla_compiler.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdla_compiler.so (0x0000007f97e23000)
        libnvmedia.so => /usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so (0x0000007f97db9000)
        libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f97d98000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f97cdf000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f97cb3000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f97c9c000)
        libcublasLt.so.10 => /usr/lib/aarch64-linux-gnu/libcublasLt.so.10 (0x0000007f95cd6000)
        libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f95cb8000)
        libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f95c75000)
        libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f95c55000)
        libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f95c35000)
        libnvtvmr.so => /usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so (0x0000007f95ba5000)
        libnvparser.so => /usr/lib/aarch64-linux-gnu/tegra/libnvparser.so (0x0000007f95b69000)
        libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f95a3d000)
        libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f95a28000)
sebawsky commented 2 years ago

This I get:

$ cd yolo/ $ ldd ../plugins/libyolo_layer.so linux-vdso.so.1 (0x00007fff56f5f000) libnvinfer.so.7 => /usr/lib/x86_64-linux-gnu/libnvinfer.so.7 (0x00007fa0e9022000) libcudart.so.10.2 => /usr/local/cuda-10.2/lib64/libcudart.so.10.2 (0x00007fa0e8da4000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa0e8a1b000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa0e8803000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa0e8412000) libcudnn.so.7 => /usr/lib/x86_64-linux-gnu/libcudnn.so.7 (0x00007fa0ccb28000) libcublas.so.10 => /usr/local/cuda-10.2/lib64/libcublas.so.10 (0x00007fa0c8871000) libmyelin.so.1 => /usr/lib/x86_64-linux-gnu/libmyelin.so.1 (0x00007fa0c805f000) libnvrtc.so.10.2 => /usr/local/cuda-10.2/lib64/libnvrtc.so.10.2 (0x00007fa0c68b2000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa0c66ae000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa0c6310000) /lib64/ld-linux-x86-64.so.2 (0x00007fa0f7fb5000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa0c60f1000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa0c5ee9000) libcublasLt.so.10 => /usr/local/cuda-10.2/lib64/libcublasLt.so.10 (0x00007fa0c4054000)

Thank @jkjung-avt

akashAD98 commented 2 years ago

@jkjung-avt im running this on ubantu 20.04 server

cd tensorrt_demos/yolo ldd ../plugins/libyolo_layer.so

    linux-vdso.so.1 (0x00007ffefbbdb000)
    libnvinfer.so.8 => /lib/x86_64-linux-gnu/libnvinfer.so.8 (0x00007f0f8e3c6000)
    libcudart.so.11.0 => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0 (0x0007f0f8e12d000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0f8df4b000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0f8df30000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f8dd3e000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f8dd31000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0f8dd2b000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0f8dbdc000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0fa66c5000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0f8dbb9000)

im getting only this files, ( actually last time i was able to convert all custom models( scaled yolo) into tensorrt on the same server with same setup ,but now im getting this issue) How should i resolve this issue? should i need to uninstall all tensorrt ,pycuda,onnx ? Thanks always @jkjung-avt

jkjung-avt commented 2 years ago

@sebawsky @akashAD98 It is OK as long as all .so files could be found. So the problems you've encountered are not due to missing dynamic libraries.

Could you try the following instead? Show me the exception logs, and I'll see how to further debug from there.

$ cd yolo/
$ python3 -c "import ctypes; ctypes.cdll.LoadLibrary('../plugins/libyolo_layer.so') "
akashAD98 commented 2 years ago

@jkjung-avt my server is off , once its switched on ill do the testing

sebawsky commented 2 years ago

@jkjung-avt I ran the indicated line and didn't get any exceptions (the answer was clean).

image

akashAD98 commented 2 years ago

@jkjung-avt same thing, there is no output, what should we do next?

jkjung-avt commented 2 years ago

@sebawsky @akashAD98 I see. So the problem is not loading of the dynamic library ("libyolo_layer.so.so"). Rather, REGISTER_TENSORRT_PLUGIN() does not work as expected ("YoloLayer_TRT" plugin creator is not registered).

The relevant source code is here. I don't know why, but will investigate more when I have time.

https://github.com/jkjung-avt/tensorrt_demos/blob/ca6ac81b4c7558122df3ff3018479412828a88a4/plugins/yolo_layer.h#L147

akashAD98 commented 2 years ago

@jkjung-avt is there any temporary solution for this? it's a little bit urgent for me, Should we try a new setup? new machine? any advice will be really helpful

akashAD98 commented 2 years ago

@jkjung-avt @sebawsky I solved this issue by uninstalling the old tensorrt version.pycuda & onxx & i did again setup with TensorRT-8.0.3.4 ,& its working fine . & again thanks for your valuable time & reply ):

akashAD98 commented 2 years ago

@jkjung-avt I have converted yolo to onnx & onnx to tensorrt, But at the time of doing testing i was not able to get detection. bcz my pycuda is installed but its showing not installed,here its log

pycudaerre.txt

jkjung-avt commented 2 years ago

@akashAD98 Please remove the whole "src/pycuda-2019.1.2" directory and re-run "install_pycuda.sh".

$ rm -rf ${HOME}/src/pycuda-2019.1.2/
$ cd ${HOME}/project/tensorrt_demos/ssd/
$ ./install_pycuda.sh
PuneethBC commented 2 years ago

I had the same issue of "RuntimeError: cannot get YoloLayer_TRT plugin creator", because the environment CUDA_VISIBLE_DEVICES was set to -1. after export CUDA_VISIBLE_DEVICES=0, it worked.

sachinkmohan commented 2 years ago

@sebawsky @akashAD98 I see. So the problem is not loading of the dynamic library ("libyolo_layer.so.so"). Rather, REGISTER_TENSORRT_PLUGIN() does not work as expected ("YoloLayer_TRT" plugin creator is not registered).

The relevant source code is here. I don't know why, but will investigate more when I have time.

https://github.com/jkjung-avt/tensorrt_demos/blob/ca6ac81b4c7558122df3ff3018479412828a88a4/plugins/yolo_layer.h#L147

@jkjung-avt , I am still confused. I can't go any further than this. Did you happen to check this?