YOLOv3 Example

[dowlingw]

Description

Looking for a sample of the YOLOv3 detector working with gocv: https://pjreddie.com/darknet/yolo/

There is a good set of sample code with both Python and C++ examples here: https://www.learnopencv.com/deep-learning-based-object-detection-using-yolov3-with-opencv-python-c/

The steps below provide a link to my my incomplete code (following the above sample) in case someone else can help/also needs this/can point me in the right direction - I'll follow up with working code as a pull request if I figure it out in the meantime.

Steps to Reproduce

Using the following test image: https://article.images.consumerreports.org/prod/content/dam/CRO%20Images%202019/Cars/January/CR-Cars-InlineHero-2019-Acura-ILX-A-Spec-f-1-19

And the following code: https://gist.github.com/dowlingw/c984def5a456b71ba5a2dd895290830e

The Mats returned from the detector seem to have Rows() and Cols() returning -1.

Sample output from the code linked above:

===
outs = 2
i = 0
out.Type() = 5
out.Rows() = -1
out.Cols() = -1
out.Size() = [1 32 320 320]
out.Total() = 3276800
out.Channels() = 1
len(data) = 3276800
...
i = 1
out.Type() = 5
out.Rows() = -1
out.Cols() = -1
out.Size() = [1 32 320 320]
out.Total() = 3276800
out.Channels() = 1
len(data) = 3276800
...

Your Environment

macOS 10.14.5 (18F132) opencv_version 4.1.0 (via homebrew) go version go1.12.4 darwin/amd64

[dowlingw]

I did see #414 but poking around the authors profile it looks like they solved the problem with cgo against the darknet libraries directly, which I'd rather avoid.

[entrehuihui]

I also hope that the project can have a specific example of yolo. I have been trying to be unsuccessful and have to directly quote C.

[dowlingw]

@entrehuihui do have a look at the gist, if you’ve used the darknet libraries you may be more familiar with the memory layout of the results which is where I’m stuck.

I’ve had to put this on hold due to family commitments but am planning to come back to it real soon!

[darrenparkinson]

One of the challenges I had was in Issue #705 where GetUnConnectedOutLayers doesn't seem to work as expected. If I hard code the names using:

fl := []string{"yolo_82", "yolo_94", "yolo_106"}
blob := gocv.BlobFromImage(frame, 1.0, image.Pt(416, 416), gocv.NewScalar(0, 0, 0, 0), true, false)
net.SetInput(blob, "data")
layerOutputs := net.ForwardLayers(fl)

I get three layers in layerOutputs with output from your code (with the car image) as follows:

===
outs = 3
i = 0
out.Type() = 5
out.Rows() = 300
out.Cols() = 85
out.Size() = [300 85]
out.Total() = 25500
out.Channels() = 1
len(data) = 25500
...
i = 1
out.Type() = 5
out.Rows() = 1200
out.Cols() = 85
out.Size() = [1200 85]
out.Total() = 102000
out.Channels() = 1
len(data) = 102000
...
i = 2
out.Type() = 5
out.Rows() = 4800
out.Cols() = 85
out.Size() = [4800 85]
out.Total() = 408000
out.Channels() = 1
len(data) = 408000
...

The Cols is 85 since my model has 80 classes in it. Apparently 0,1,2,3 are centerX, centerY, width and height respectively?

I think I'm then able to get the rows as follows to identify matches (there is probably a more appropriate way, but appears to work for now):

for _, output := range layerOutputs {
    data, err := output.DataPtrFloat32()
    if err != nil {
        log.Fatal(err)
    }
    for i := 0; i < output.Total(); i += output.Cols() {
    rowCount++
    row := data[i : i+output.Cols()]
    scores := row[5:]
    _, max := findMinMax(scores)
    if max > 0.6 {
        fmt.Println(scores, max)
    }
    }
}

I get 11 matches on the image using 0.6 as the threshold. Unfortunately, the class IDs don't seem to match what I'd expect, so I'm definitely doing something wrong, just not sure what yet.

Then there is the issue of non-maxima suppression(?) which is supposed to remove overlapping boxes given a threshold. OpenCV has an NMSBoxes function but I don't see an equivalent function in GoCV -- might log a separate issue on that.

Not sure if this is progress or not tbh 🤷

[florianpinel]

@darrenparkinson use a scale factor of 1/255: blob := gocv.BlobFromImage(frame, 1.0 / 255.0, image.Pt(416, 416), gocv.NewScalar(0, 0, 0, 0), true, false) If this returns an error, convert your frame first: frame.ConvertTo(frame, gocv.MatTypeCV32F)