opencv's dnn module not working with person-detection-retail-0013

[n-mam]

Hi

I had a working opencv dnn setup with MobileNetSSD_deploy caffemodel for person detection. I successfully compiled opencv (master) with openvino(2020.4.287). However when I replace the cafemodels with FP16 person-detection-retail-0013 model files, I do not get any detections. The confidence levels seems to be too low (0.01xy) Could you please advise on what could be the issue here ? Is there anything else that needs to be done besides building opencv with opnvino enabled ? also where could I find a working dnn sample with openvino pretrained models ? below I have pasted the relevant portion of my code. Also at startup I see these messages as part of enabling the backend to IE

also the same model files work fine with openvino's pedestrian tracking demo

[E:] [BSL] found 0 ioexpander device
IE backend and cpu target enabled for inference.

    ObjectDetector(const std::string& target) :
     CDetector("person-detection-retail-0013/FP16/person-detection-retail-0013.bin", "person-detection-retail-0013/FP16/person-detection-retail-0013.xml") 
     //CDetector("MobileNetSSD_deploy.prototxt", "MobileNetSSD_deploy.caffemodel") 
    {
      iTarget = target;
    }

    virtual std::vector<cv::Rect2d> Detect(cv::Mat& frame) override
    {
      std::vector<cv::Rect2d> out;

      cv::Mat resized;

      cv::resize(frame, resized, cv::Size(300, 300));

      cv::Mat inputBlob = cv::dnn::blobFromImage(
        resized,
        0.007843f,
        cv::Size(300, 300),
        cv::Scalar(127.5, 127.5, 127.5),
        false, 
        false);

      iNetwork.setInput(inputBlob);

      cv::Mat detection = iNetwork.forward();

      cv::Mat detectionMat(detection.size[2], detection.size[3], CV_32F, detection.ptr<float>());

      for (int i = 0; i < detectionMat.rows; i++)
      {
        float confidence = detectionMat.at<float>(i, 2);

        if (confidence > 0.6)
        {
          int idx, x1, y1, x2, y2;

          idx = static_cast<int>(detectionMat.at<float>(i, 1));

          if (iObjectClass[idx] == iTarget)
          {
            x1 = static_cast<int>(detectionMat.at<float>(i, 3) * frame.cols);
            y1 = static_cast<int>(detectionMat.at<float>(i, 4) * frame.rows);
            x2 = static_cast<int>(detectionMat.at<float>(i, 5) * frame.cols);
            y2 = static_cast<int>(detectionMat.at<float>(i, 6) * frame.rows);
            out.emplace_back(cv::Point(x1, y1), cv::Point(x2, y2));
            //std::cout << "Object(" + iObjectClass[idx] + ") detected at " << x1 << "," << y1 << "[" << x2 - x1 << "," << y2 - y1 << "]\n";
          }      
        }
      }

      return out;
    }

[n-mam]

further information: on using the original mobilenet ssd model files from inside openvino setupvars env I get the below errors:

IE backend and cpu target enabled for inference.
[E:] [BSL] found 0 ioexpander device
[E:] [BSL] found 0 ioexpander device
OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.5.0-pre) Error: Unspecified error (> Failed to initialize Inference Engine backend (device = GPU): Program buil
d failed:
>
>
> ) in cv::dnn::InfEngineNgraphNet::initPlugin, file ..\modules\dnn\src\ie_ngraph.cpp, line 715

[n-mam]

I see that face detection using opencv dnn api and face-detection-adas-0001 model works fine. However all the samples which use person-detection-retail-0013 make exclusive use of use of openvino api. I didnt see any sample which makes use of opencv's dnn api with person-detection-retail-0013 model. Do we have some docs around which api level (openvino vs dnn) works with which models ?

[jgespino]

Hi @n-mam

That is correct, the samples and demos from OpenVINO toolkit make use of the Inference Engine.

For OpenCV DNN, we used to have a Python sample in the OpenVINO Raspbian OS installation guide. I did a quick test with the OpenVINO toolkit 2021.1 release using the code below and did not see any issues running inference with person-detection-retail-0013.

import cv2 as cv
# Load the model.
net = cv.dnn.readNet('person-detection-retail-0013.xml',
                     'person-detection-retail-0013.bin')
# Specify target device.
net.setPreferableTarget(cv.dnn.DNN_TARGET_CPU)
# Read an image.
frame = cv.imread('image.jpg')
if frame is None:
    raise Exception('Image not found!')
# Prepare input blob and perform an inference.
blob = cv.dnn.blobFromImage(frame, size=(672, 384), ddepth=cv.CV_8U)
net.setInput(blob)
out = net.forward()
# Draw detected faces on the frame.
for detection in out.reshape(-1, 7):
    confidence = float(detection[2])
    xmin = int(detection[3] * frame.shape[1])
    ymin = int(detection[4] * frame.shape[0])
    xmax = int(detection[5] * frame.shape[1])
    ymax = int(detection[6] * frame.shape[0])
    if confidence > 0.5:
        cv.rectangle(frame, (xmin, ymin), (xmax, ymax), color=(0, 255, 0))
# Save the frame to an image file.
cv.imwrite('out.png', frame)

@dkurt Do you know if we have additional C/C++ samples?

Regards, Jesus

[n-mam]

Great! it is working now... many thanks for the python sample. Changed the image size and the ddepth values to blobFromImage. also my earlier MobileNetSSD was class based detection and I forgot to remove the object class check while using openvino model. Thanks again.

[dkurt]

@dkurt Do you know if we have additional C/C++ samples?

All the OpenCV samples are located in OpenCV repo. Take a look at https://github.com/opencv/opencv/blob/master/samples/dnn/object_detection.cpp

[n-mam]

@dkurt thanks. Not sure if this covers all aspects of how the results are to be interpreted. For example check this post which highlights similar issues while integrating openvino models with dnn api. Based on the discussion in this thread I was able to get the age part working reliably but I am not sure how to extract the gender part... Could you please advise ?

      auto blob = cv::dnn::blobFromImage(frame, 1, cv::Size(62, 62));
      iNetwork.setInput(blob);
      std::vector<cv::Mat> out;
      std::vector<cv::String> layers{"age_conv3", "prob"};
      iNetwork.forward(out, layers);
      float age = out[0].at<float>(0) * 100; ---> this works okay
      float gprob = out[0].at<float>(1);  ---> this always returns `4.13958e-41` irrespective of M/F

some pythons samples on github have:

res.isMale = genderBlob->buffer().as<float*>()[1] > 0.5;

gender = "Female" if np.argmax(pred["prob"][0]) == 0 else "Male"

[dkurt]

Please try this instead:

std::vector<cv::String> layers = iNetwork.getUnconnectedOutLayersNames();
iNetwork.forward(out, layers);

[n-mam]

okay thanks. For females this change gives out[0].at(1) a value of 0 and for males it comes out as as 7.79122e-43. Would this be correct ? If yes, then which of the 2 earlier python checks (reproduced below) would be more representative of the M/F differentiation ?

value of out[0].at(1) > 0.5 as being male res.isMale = genderBlob->buffer().as<float*>()[1] > 0.5;

or value of out[0].at(1) == 0 as being female gender = "Female" if np.argmax(pred["prob"][0]) == 0 else "Male"

[n-mam]

I guess I'll incorporate the age and gender processing as done in the openvino sample based on male female scores Thanks again for your pointers. I'll close this now.