Error when inference YOLOv4 engine model - [executionContext.cpp::enqueueV2::520] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueV2::520, condition: !mEngine.hasImplicitBatchDimension()?

[aidevmin]

My project based on your repo with commit SHA - 68f762d5bdeae7ac3458529bfe6fed72714336ca https://github.com/marcoslucianops/DeepStream-Yolo/tree/68f762d5bdeae7ac3458529bfe6fed72714336ca I generated int8 engine model and use repo https://github.com/Linaom1214/TensorRT-For-YOLO-Series/blob/main/trt.py to evaluate engine model, but I got an error

[executionContext.cpp::enqueueV2::520] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueV2::520, condition: !mEngine.hasImplicitBatchDimension()

This error is not happen with lasted version of your repo, but I need to use older version. How to fix this @marcoslucianops? Thanks

[aidevmin]

@marcoslucianops I fixed it. But output of generated int8 engine is not correct. Yolov4 engine output has 4 elements:

• first element: number of detections data[0]
• second element: bounding boxes data[1]
• third element: score data[2]
• fourth element: class index data[3]

  INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 5
  0   INPUT  kFLOAT data            3x608x608       
  1   OUTPUT kFLOAT num_detections  1               
  2   OUTPUT kFLOAT detection_boxes 22743x4         
  3   OUTPUT kFLOAT detection_scores 22743           
  4   OUTPUT kFLOAT detection_classes 22743  

  

You can see the above debug image: class index is not integer (4.2 e-44). I check many images max(class index) is float < 1 (it should be 79). And number of detections data[0] < 0. I used version https://github.com/marcoslucianops/DeepStream-Yolo/tree/68f762d5bdeae7ac3458529bfe6fed72714336ca for YOLOv4. Where can I wrong?

[marcoslucianops]

The new version of the repo isn't compatible with the old files (libs, texts, engines, etc). I recommend you to use the updated files.

[aidevmin]

@marcoslucianops I use old version with old files but output values seem not correct

[marcoslucianops]

I generated int8 engine model and use repo https://github.com/Linaom1214/TensorRT-For-YOLO-Series/blob/main/trt.py to evaluate engine model, but I got an error

You can't evaulate the engines from this repo in other codes because it's generated specifically for the network and output I set in this repo for the DeepStream.

[aidevmin]

@marcoslucianops Yeah. I know the output format is different and I has changed source code. Until now I only get the output of engine model to check the correctness. As in the above image, data is the raw output of engine model when feeding image. I checked length of data, shape of data components, and its values. data[3] is class index from 0 to 79, but I checked for images its values are not correct.

[marcoslucianops]

I'm not sure how the old code works.

You can see the above debug image: class index is not integer (4.2 e-44). I check many images max(class index) is float < 1 (it should be 79). And number of detections data[0] < 0.

The num_detections have the number of valid detections. You need to check the bboxes, scores and classes from 0 to the num_detections[0] value. If the num_detections[0] is 0, it means that there's no detection in the image, and the values in the array are only the allocated memory (they aren't valid).

[marcoslucianops]

4 OUTPUT kFLOAT detection_classes 22743

The detection_classes output is a kFLOAT array. You need to cast it as float and convert to int after the cast.

[aidevmin]

Thanks. For Darknet model, output from engine model is same as output from Darknet model or you modified it? I see you did many custom layers (file .cpp). For what are these .cpp files? I see that Nvidia offical repo doesn't have any .cpp filea.

[marcoslucianops]

For Darknet model, output from engine model is same as output from Darknet model or you modified it?

It has the post-processing in the engine.

I see that Nvidia offical repo doesn't have any .cpp filea

This repo is an improvement of the NVIDIA YOLO. Here you will have support for more models and easier implementation.

[aidevmin]

It has the post-processing in the engine.

It may be the reason there is a strange value when running inference by using Tensorrt Python API. Could you point me which file you custom to do postprocessing in engine?

As my understanding, Deepstream generate engine model, run inference on image, and output of engine will be passed into nvdsparsebbox_yolo.cpp and finnaly passed into NMS? Is it right?

You mentioned It has the post-processing in the engine. Which post processing do you mean? Is it different from post-processing as I mentioned above?

[marcoslucianops]

if you use the latest commit, the output will be

(float) boxes = outputLayersInfo[0];
(float) scores = outputLayersInfo[1];
(float) classes = outputLayersInfo[2];

(int) outputSize = boxes.inferDims.d[0]; // first dimension size of the layer

for (uint b = 0; b < outputSize; ++b) {
  float maxProb = scores[b];  // best detection score
  int maxIndex = (int) classes[b];  // class with the best detection score

  float bxc = boxes[b * 4 + 0];  // x_center
  float byc = boxes[b * 4 + 1];  // y_center
  float bw = boxes[b * 4 + 2];  // width
  float bh = boxes[b * 4 + 3];  // height
}

[aidevmin]

Yeah, thanks. I have read these code for parsing bboxes. This processing is not included in engine. Honesty, I didn't find the reason why output from Deepstream generated engine model when running by using Tensorrt API is strange.

[marcoslucianops]

The post-processing included in the engine is the nvdsinfer_custom_impl_Yolo/yoloForward.cu / nvdsinfer_custom_impl_Yolo/yoloForward_nc.cu

[marcoslucianops]

If possible, send to my email more details about what you are doing and I will try to help you.

[aidevmin]

@marcoslucianops Thank you so much. Please check email.