fatbinary_section.h: No such file or directory

[lyjdwh]

When I make this, I get this error. I use cuda 10.2, cudnn 7.6.5, tensorrt 7.0.0.11

-- Found TensorRT headers at /root/TensorRT-7.0.0.11/include
-- Find TensorRT libs at /root/TensorRT-7.0.0.11/lib/libnvinfer.so;/root/TensorRT-7.0.0.11/lib/libnvparsers.so;/root/TensorRT-7.0.0.11/lib/libnvinfer_plugin.so
-- WITH_DEEPSTREAM: false
CMake Warning at CMakeLists.txt:107 (message):
  Detected CUDA version is < 11.0.  SM80 not supported.

-- GPU_ARCHS is not defined. Generating CUDA code for default SMs: 35;53;61;70;75
-- The CUDA compiler identification is unknown
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - failed
-- Check for working CUDA compiler: /usr/bin/nvcc
-- Check for working CUDA compiler: /usr/bin/nvcc - broken
CMake Error at /usr/local/share/cmake-3.20/Modules/CMakeTestCUDACompiler.cmake:52 (message):
  The CUDA compiler

    "/usr/bin/nvcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /root/amirstan_plugin/build/CMakeFiles/CMakeTmp

    Run Build Command(s):/usr/bin/make -f Makefile cmTC_10be4/fast && /usr/bin/make  -f CMakeFiles/cmTC_10be4.dir/build.make CMakeFiles/cmTC_10be4.dir/build
    make[1]: Entering directory '/root/amirstan_plugin/build/CMakeFiles/CMakeTmp'
    Building CUDA object CMakeFiles/cmTC_10be4.dir/main.cu.o
    /usr/bin/nvcc    -Xcompiler -Wno-deprecated-declarations   -c /root/amirstan_plugin/build/CMakeFiles/CMakeTmp/main.cu -o CMakeFiles/cmTC_10be4.dir/main.cu.o
    fatbinary warning : option 'cuda' has been deprecated
    In file included from /tmp/tmpxft_0000d462_00000000-5_main.cudafe1.stub.c:6:0,
                     from tmpxft_0000d462_00000000-5_main.cudafe1.stub.c:1:
    /tmp/tmpxft_0000d462_00000000-2_main.fatbin.c:2:31: fatal error: fatbinary_section.h: No such file or directory
     #include "fatbinary_section.h"
                                   ^
    compilation terminated.
    CMakeFiles/cmTC_10be4.dir/build.make:77: recipe for target 'CMakeFiles/cmTC_10be4.dir/main.cu.o' failed
    make[1]: *** [CMakeFiles/cmTC_10be4.dir/main.cu.o] Error 1
    make[1]: Leaving directory '/root/amirstan_plugin/build/CMakeFiles/CMakeTmp'
    Makefile:127: recipe for target 'cmTC_10be4/fast' failed
    make: *** [cmTC_10be4/fast] Error 2

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  src/plugin/CMakeLists.txt:6 (enable_language)

-- Configuring incomplete, errors occurred!
See also "/root/amirstan_plugin/build/CMakeFiles/CMakeOutput.log".
See also "/root/amirstan_plugin/build/CMakeFiles/CMakeError.log".

Do you have any idea? Thanks so much!

[grimoire]

I am not sure.

I found a similar error in other repos with solution, see if it can help you.

[grimoire]

And tensorrt 7.0 is not recommended. It might meet memory leak on some device.

[lyjdwh]

This solution works! Thanks so much!

But when I use the converted tensorrt model, I get unexpected key in source state_dict: engine, input_names, output_names. I use mmdet 2.13.0

from mmdet.apis import inference_detector, init_detector
import os

# Choose to use a config and initialize the detector
config = '/root/mmdetection/zyys_great_experiments/config_ikcest.py'
# Setup a checkpoint file to load
checkpoint = '/root/mmdetection/work_dirs/config_ikcest/latest_tensorrt.pth'
# initialize the detector
model = init_detector(config, checkpoint, device='cuda:1')

# Use the detector to do inference
imgs = [os.path.join("/root/databases/ikcest/test_dataset", file) for file in os.listdir("/root/databases/ikcest/test_dataset")]
for i in range(20):
    test_imgs = imgs[i*20: (i+1)*20]
    result = inference_detector(model, test_imgs)

[grimoire]

If you are using mmdetection-to-tensorrt to convert model. The inference interface is in here, not the official one. here is the demo. I haven't tested the repo on mmdetection2.13，if you find any problem, please feel free to report here.

[lyjdwh]

I changed to mmdet 2.10, but I still have some error. Do you known why?

(open-mmlab) root@bms--Ubuntu:~/mmdetection/zyys_great_experiments# python test_tensorrt.py
0
[TensorRT] ERROR: Parameter check failed at: engine.cpp::setBindingDimensions::1046, condition: profileMinDims.d[i] <=
dimensions.d[i]
[TensorRT] ERROR: Parameter check failed at: engine.cpp::resolveSlots::1228, condition: allInputDimensionsSpecified(routine)
Traceback (most recent call last):
  File "test_tensorrt.py", line 27, in <module>
    result = inference_detector(model, test_img, config, "cuda:1")
  File "/root/mmdetection-to-tensorrt/mmdet2trt/apis/inference.py", line 48, in inference_detector
    result = model(tensor)
  File "/root/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/torch2trt_dynamic/torch2trt_dynamic/torch2trt_dynamic.py", line 478, in forward
    shape = tuple(self.context.get_binding_shape(idx))
ValueError: __len__() should return >= 0

[grimoire]

This might help you with your problem link