search

Linaom1214 / TensorRT-For-YOLO-Series

tensorrt for yolo series (YOLOv10,YOLOv9,YOLOv8,YOLOv7,YOLOv6,YOLOX,YOLOv5), nms plugin support
901 stars 152 forks source link

c++ Error end2end #63

Closed MyraBaba closed 1 year ago

MyraBaba commented 1 year ago

Hi,

I converted my model to pg - onnx and than trt as explained. python trt is working but c++ end2end has below error:

./yolo -model_path /data/dProjects/yolov7/yolov7-tiny-nms.trt -image_path /data/dProjects/yolov7/tensorrt-python/cpp/end2end/build/testCar.jpg

test image is : 2560x1440

model size: 144312333 Registered plugin creator - ::GridAnchor_TRT version 1 Registered plugin creator - ::GridAnchorRect_TRT version 1 Registered plugin creator - ::NMS_TRT version 1 Registered plugin creator - ::Reorg_TRT version 1 Registered plugin creator - ::Region_TRT version 1 Registered plugin creator - ::Clip_TRT version 1 Registered plugin creator - ::LReLU_TRT version 1 Registered plugin creator - ::PriorBox_TRT version 1 Registered plugin creator - ::Normalize_TRT version 1 Registered plugin creator - ::ScatterND version 1 Registered plugin creator - ::RPROI_TRT version 1 Registered plugin creator - ::BatchedNMS_TRT version 1 Registered plugin creator - ::BatchedNMSDynamic_TRT version 1 Registered plugin creator - ::BatchTilePlugin_TRT version 1 Registered plugin creator - ::FlattenConcat_TRT version 1 Registered plugin creator - ::CropAndResize version 1 Registered plugin creator - ::CropAndResizeDynamic version 1 Registered plugin creator - ::DetectionLayer_TRT version 1 Registered plugin creator - ::EfficientNMS_TRT version 1 Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1 Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1 Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1 Registered plugin creator - ::ProposalDynamic version 1 Registered plugin creator - ::Proposal version 1 Registered plugin creator - ::ProposalLayer_TRT version 1 Registered plugin creator - ::PyramidROIAlign_TRT version 1 Registered plugin creator - ::ResizeNearest_TRT version 1 Registered plugin creator - ::Split version 1 Registered plugin creator - ::SpecialSlice_TRT version 1 Registered plugin creator - ::InstanceNormalization_TRT version 1 Registered plugin creator - ::InstanceNormalization_TRT version 2 Registered plugin creator - ::CoordConvAC version 1 Registered plugin creator - ::DecodeBbox3DPlugin version 1 Registered plugin creator - ::GenerateDetection_TRT version 1 Registered plugin creator - ::MultilevelCropAndResize_TRT version 1 Registered plugin creator - ::MultilevelProposeROI_TRT version 1 Registered plugin creator - ::NMSDynamic_TRT version 1 Using cuDNN as a tactic source TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2 Deserialization required 537903 microseconds. 3: Cannot find binding of given name: num 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: boxes 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: scores 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: classes 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) Using cublasLt as a tactic source Using cuDNN as a tactic source TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2 Total per-runner device persistent memory is 2708992 Total per-runner host persistent memory is 276464 Allocated activation device memory of size 73804800 img : /data/dProjects/yolov7/tensorrt-python/cpp/end2end/build/testCar.jpg 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 12ms

Screen Shot 2022-10-15 at 20 37 48

converted onns input name is "images"

Linaom1214 commented 1 year ago

Hi,

I converted my model to pg - onnx and than trt as explained. python trt is working but c++ end2end has below error:

./yolo -model_path /data/dProjects/yolov7/yolov7-tiny-nms.trt -image_path /data/dProjects/yolov7/tensorrt-python/cpp/end2end/build/testCar.jpg

test image is : 2560x1440

model size: 144312333 Registered plugin creator - ::GridAnchor_TRT version 1 Registered plugin creator - ::GridAnchorRect_TRT version 1 Registered plugin creator - ::NMS_TRT version 1 Registered plugin creator - ::Reorg_TRT version 1 Registered plugin creator - ::Region_TRT version 1 Registered plugin creator - ::Clip_TRT version 1 Registered plugin creator - ::LReLU_TRT version 1 Registered plugin creator - ::PriorBox_TRT version 1 Registered plugin creator - ::Normalize_TRT version 1 Registered plugin creator - ::ScatterND version 1 Registered plugin creator - ::RPROI_TRT version 1 Registered plugin creator - ::BatchedNMS_TRT version 1 Registered plugin creator - ::BatchedNMSDynamic_TRT version 1 Registered plugin creator - ::BatchTilePlugin_TRT version 1 Registered plugin creator - ::FlattenConcat_TRT version 1 Registered plugin creator - ::CropAndResize version 1 Registered plugin creator - ::CropAndResizeDynamic version 1 Registered plugin creator - ::DetectionLayer_TRT version 1 Registered plugin creator - ::EfficientNMS_TRT version 1 Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1 Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1 Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1 Registered plugin creator - ::ProposalDynamic version 1 Registered plugin creator - ::Proposal version 1 Registered plugin creator - ::ProposalLayer_TRT version 1 Registered plugin creator - ::PyramidROIAlign_TRT version 1 Registered plugin creator - ::ResizeNearest_TRT version 1 Registered plugin creator - ::Split version 1 Registered plugin creator - ::SpecialSlice_TRT version 1 Registered plugin creator - ::InstanceNormalization_TRT version 1 Registered plugin creator - ::InstanceNormalization_TRT version 2 Registered plugin creator - ::CoordConvAC version 1 Registered plugin creator - ::DecodeBbox3DPlugin version 1 Registered plugin creator - ::GenerateDetection_TRT version 1 Registered plugin creator - ::MultilevelCropAndResize_TRT version 1 Registered plugin creator - ::MultilevelProposeROI_TRT version 1 Registered plugin creator - ::NMSDynamic_TRT version 1 Using cuDNN as a tactic source TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2 Deserialization required 537903 microseconds. 3: Cannot find binding of given name: num 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: boxes 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: scores 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: classes 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) Using cublasLt as a tactic source Using cuDNN as a tactic source TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2 Total per-runner device persistent memory is 2708992 Total per-runner host persistent memory is 276464 Allocated activation device memory of size 73804800 img : /data/dProjects/yolov7/tensorrt-python/cpp/end2end/build/testCar.jpg 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 12ms

Screen Shot 2022-10-15 at 20 37 48

converted onns input name is "images"

''' auto in_dims = engine->getBindingDimensions(engine->getBindingIndex("image_arrays")); iH = in_dims.d[2]; iW = in_dims.d[3]; in_size = 1; for (int j = 0; j < in_dims.nbDims; j++) { in_size = in_dims.d[j]; } auto out_dims1 = engine->getBindingDimensions(engine->getBindingIndex("num")); out_size1 = 1; for (int j = 0; j < out_dims1.nbDims; j++) { out_size1 = out_dims1.d[j]; } auto out_dims2 = engine->getBindingDimensions(engine->getBindingIndex("boxes")); out_size2 = 1; for (int j = 0; j < out_dims2.nbDims; j++) { out_size2 = out_dims2.d[j]; } auto out_dims3 = engine->getBindingDimensions(engine->getBindingIndex("scores")); out_size3 = 1; for (int j = 0; j < out_dims3.nbDims; j++) { out_size3 = out_dims3.d[j]; } auto out_dims4 = engine->getBindingDimensions(engine->getBindingIndex("classes")); ''' modify the binging name same as your onnx model.

MyraBaba commented 1 year ago

@Linaom1214 Hi,

Already changed the binding input name: 

` engine = runtime->deserializeCudaEngine((void*)&buf[0], mdsize, nullptr);

  auto in_dims = engine->getBindingDimensions(engine->getBindingIndex("images"));
  auto out_dims1 = engine->getBindingDimensions(engine->getBindingIndex("num"));
  auto out_dims2 = engine->getBindingDimensions(engine->getBindingIndex("boxes"));
  auto out_dims3 = engine->getBindingDimensions(engine->getBindingIndex("scores"));
  auto out_dims4 = engine->getBindingDimensions(engine->getBindingIndex("classes"));
  context = engine->createExecutionContext();`

Error :

Using cublasLt as a tactic source Using cuDNN as a tactic source TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2 Deserialization required 535891 microseconds. 3: Cannot find binding of given name: num 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: boxes 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: scores 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) 3: Cannot find binding of given name: classes 3: [engine.cpp::getBindingDimensions::1307] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::getBindingDimensions::1307, condition: bindIndex >= 0 && bindIndex < getNbBindings() ) Using cublasLt as a tactic source Using cuDNN as a tactic source TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2 Total per-runner device persistent memory is 2708992 Total per-runner host persistent memory is 276464 Allocated activation device memory of size 73804800 img : testCar.jpg 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed. ) 10ms

Linaom1214 commented 1 year ago

same as your onnx model.

you should keep the code as follow:

engine = runtime->deserializeCudaEngine((void*)&buf[0], mdsize, nullptr);

auto in_dims = engine->getBindingDimensions(engine->getBindingIndex("images"));
auto out_dims1 = engine->getBindingDimensions(engine->getBindingIndex("num_dets"));
auto out_dims2 = engine->getBindingDimensions(engine->getBindingIndex("det_boxes"));
auto out_dims3 = engine->getBindingDimensions(engine->getBindingIndex("det_scores"));
auto out_dims4 = engine->getBindingDimensions(engine->getBindingIndex("det_classes"));
context = engine->createExecutionContext();
MyraBaba commented 1 year ago

@Linaom1214 How I missed this ? I assume that only the input name changes is enough according to the README / manual.

is there any significant speed difference between python / cpp tensorrt version?

Best / Thanks

Linaom1214 commented 1 year ago

thanks I will update the README file