Issues with chapters 15 and 16

[Windwoes]

Date on version you are commenting on

May 1 2022

Section you are commenting on

15 and 16

Comments

• Rumble support was actually added in SDK v7.0, not 6.0 :')
• 16.1: "Luckily for OpenCV, there is a project called OpenCV." Typo here I think. Also in the next sentence "It is very common for people to forget to copy the openCV library onto the Android device" It would probably be better to specify that it's the native shared library (.so) file specifically that you're talking about.
• 16.2: "In our start, we stop the streaming" - stopStreaming() is a blocking call, would suggest closeCameraDeviceAsync() instead.
• 16.2.2: " (It is called for EVERY frame in the video, so it needs to do as little as possible.)" - possibly a bit misleading - taking too long inside processFrame() will not cause a "lag stackup" or anything. The stream is double buffered so if you miss one inter-frame deadline the next frame is still stored for you to work on immediately, but beyond that the frames are simply dropped.
• 16.6: In your FirstPipelineRevised you are releasing hsvMat and input every time around processFrame; this is not required. You have set up hsvMat as an instance variable rather than a local variable, so the framebuffer will be preserved and re-used across calls to processFrame; this is good and reduced memory pressure compared to allocing and freeing it every time. Also, you should never call release() on input.

[alan412]

• Wrong version of SDK - fixed
• 16.1 comment - fixed
• 16.2 - we actually use stopStreaming because it doesn't close the connection to your camera which allows you to start it again later if needed with less lag. Is there a non-blocking call that stops the streaming but doesn't close the device?
• 16.2.2 - I modified the text. Please let me know what you think
• 16.6 - If you remove either of those release() then the pipeline has a memory leak. You can see it by running them in the simulator and checking memory usage and watch it climb.
  • For the input mat, my understanding (which may not be correct) is that if you return a different mat than the one that was passed to you then you need to release the one that was passed to you and the one you return will continue in the processing.
  • For the HSV mat, you need to release it since it is being filled by the cvtColor call. (It is possible and even likely that there is something I don't understand here. Most of my java knowledge is as an FTC coach or FRC mentor.)

Also, if you'll give me your name and any associated FTC team then I'll be happy to add you to the Credits. I really appreciate your taking the time to give feedback!!

[Windwoes]

we actually use stopStreaming because it doesn't close the connection to your camera which allows you to start it again later if needed with less lag. Is there a non-blocking call that stops the streaming but doesn't close the device?

Hmm, EOCV does not currently have a non-blocking call to stop streaming, no. I suppose you could make a new thread and call it from that. Not exactly best practice though.... probably something I could support in the next release of EOCV though.

I modified the text. Please let me know what you think

Looks better :')

For the input mat, my understanding (which may not be correct) is that if you return a different mat than the one that was passed to you then you need to release the one that was passed to you and the one you return will continue in the processing.

Well, first things first, your example code actually crashes on EOCV on real hardware because releasing input() breaks some internal logic that checks to make sure a sane Mat size was returned for display on the viewport. (It will throw an exception complaining that "User pipeline returned frame of unexpected size"). The input Mat that is delivered to your Pipeline is always the same object - the framebuffer is static and is re-used. There is no memory leak from not releasing it.

For the HSV mat, you need to release it since it is being filled by the cvtColor call.

Because it's an instance variable and not a local variable, the same chunk of memory is being filled each time so there is no leak. Also, after removing the input.release() call to allow the code to run on real hardware, I'm actually getting a memory leak even with the call to release the HSV mat.

I think it's leaking due to Core.split replacing the old Mats in the arraylist without releasing them.

Here's a modified version of your pipeline which runs on real hardware and does not leak memory and does not need to call .release(). If this causes a memory leak in EOCV-Sim then that's a bug in the simulator that you should report :)

public class FirstPipelineRevised extends OpenCvPipeline
{
    public Scalar nonSelectedColor = new Scalar(0,255,0);
    public Scalar selectedColor = new Scalar(0,0,255);

    public Rect rect1 = new Rect(110, 42, 40, 40);
    public Rect rect2 = new Rect(160, 42, 40, 40);
    public Rect rect3 = new Rect(210, 42, 40, 40);
    public int selectedRect = 0;

    Mat hsvMat = new Mat();
    Mat destMat = new Mat();
    Mat detectionMat = new Mat();

    @Override
    public Mat processFrame(Mat input)
    {
        Imgproc.cvtColor(input, hsvMat, Imgproc.COLOR_RGB2HSV);
        Core.extractChannel(hsvMat, detectionMat, 1);
        Imgproc.cvtColor(detectionMat, destMat, Imgproc.COLOR_GRAY2RGB);

        drawRectangles(destMat);

        return destMat;
    }

    public void drawRectangles(Mat input)
    {
        Imgproc.rectangle(input, rect1, nonSelectedColor);
        Imgproc.rectangle(input, rect2, nonSelectedColor);
        Imgproc.rectangle(input, rect3, nonSelectedColor);

        switch (selectedRect) {
            case 1:
                Imgproc.rectangle(input, rect1, selectedColor);
                break;
            case 2:
                Imgproc.rectangle(input, rect2, selectedColor);
                break;
            case 3:
                Imgproc.rectangle(input, rect3, selectedColor);
                break;
        }
    }
}

---

The thing with memory leaking is that you want to avoid allocating new Mats in processFrame(). If must do it, then you have to release them. But if you allocate Mats outside processFrame and simply reference them inside, the chunk of memory is being re-used and there is no leak.

[alan412]

Thank you very much!! I didn't know about extractChannel - that is a MUCH cleaner way of doing it. What is your name so I can put you in the Credits?

[Windwoes]

Michael - 4634 alum