ZEDMat to OpenCvMat - Githubissues

As the title says I'm trying to convert a ZEDMat (or at least the left cameras texture) to an OpenCv Mat. I've found some people who have done it in C++ however the method they used doesn't seem to be available in the C# version of OpenCv that I am using.

What I'm currently trying to do is get the ZEDMat via ZEDCamera.RetrieveImage, pull the byte array from it, turn that into a Unity Texture2D (probably don't need this step but it seems like the easiest way to mix the two libraries), and then turn that Texture2D into an OpenCv Mat. Through tutorials I have been able to handle the Texture2D to OpenCv part, and after extensive trial and error found that I am getting a ZEDMat with data (took a while to find the write function and figure out that the texture was probably stored on the GPU and unavailable to the ZEDCamera.GetTexture function). Now I am just trying to convert the ZEDMat to a Texture2D. The documentation in the code says this can be done easily by getting the pointer to the ZEDMat byte stream, using Marshal.Copy to copy it to a byte array, and then loading that into a Texture2D, but obviously I am doing something wrong. When I've tried the suggested solution I just get a Unity 8x8 question mark image in the texture. I have no idea what is wrong or what to do, looking at the byte data it seems valid but I have no idea what I am really looking for anyway (basically no DEADBEEF or all 0s).

Here's where I decided to call it a night:

byte[] matrixBytes = new byte[zedMat.GetPixelBytes() * zedMat.GetWidth() * zedMat.GetHeight()];
System.Runtime.InteropServices.Marshal.Copy(zedMat.GetPtr(), matrixBytes, 0, matrixBytes.Length);

Texture2D result = new Texture2D(zedMat.GetWidth(), zedMat.GetHeight(), TextureFormat.RGBA32, false);
result.LoadImage(matrixBytes);
result.Apply();

System.IO.File.WriteAllBytes("testFile.png", result.EncodeToPNG()); // Check file data in windows

So to convert the ZED image to a Texture2D, I recommend taking a look at this code as reference (just note their mistake in the while loop condition, which they mention in a subsequent comment).

At first glance, it does appear that your issue is trying to copy data from the GPU. You can specify which memory type (CPU or GPU) the ZEDMat keeps its data on in its constructor, and also when you call RetrieveImage.

However, if your target is an OpenCV Mat, it's much simpler to convert it directly (though likely tougher to figure it out since there's not documentation on it). The trick is that once you have a CVMat formatted with the correct format and of the correct size, you can use Utils.copyToMat and copy the data from the ZEDMat's IntPtr directly to the OpenCV Mat.

So usually you have a function to convert the ZEDMat.MAT_TYPE of a material to an OpenCV CvType. This one takes a ZEDMat and the ZEDMat type, and returns a properly sized and formatted OpenCV Mat (without actually copying over the data yet):

    private static Mat SLMat2CVMat(sl.ZEDMat zedmat, ZEDMat.MAT_TYPE zedmattype)
    {
        int cvmattype = -1;
        switch (zedmattype)
        {
            case ZEDMat.MAT_TYPE.MAT_32F_C1:
                cvmattype = CvType.CV_32FC1;
                break;
            case ZEDMat.MAT_TYPE.MAT_32F_C2:
                cvmattype = CvType.CV_32FC2;
                break;
            case ZEDMat.MAT_TYPE.MAT_32F_C3:
                cvmattype = CvType.CV_32FC3;
                break;
            case ZEDMat.MAT_TYPE.MAT_32F_C4:
                cvmattype = CvType.CV_32FC4;
                break;
            case ZEDMat.MAT_TYPE.MAT_8U_C1:
                cvmattype = CvType.CV_8UC1;
                break;
            case ZEDMat.MAT_TYPE.MAT_8U_C2:
                cvmattype = CvType.CV_8UC2;
                break;
            case ZEDMat.MAT_TYPE.MAT_8U_C3:
                cvmattype = CvType.CV_8UC3;
                break;
            case ZEDMat.MAT_TYPE.MAT_8U_C4:
                cvmattype = CvType.CV_8UC4;
                break;
        }

        Mat cvmat = new Mat(zedmat.GetHeight(), zedmat.GetWidth(), cvmattype);

        return cvmat;
    }

What I usually do is subscribe an event to ZEDManager.OnGrab that handles the RetrieveImage call, and then deploys another event that my OpenCV-handling scripts subscribe to. Doing it this way also makes sure that you start processing a new frame close to when it's available, avoiding latency like you might get if running in Update(). My event looks like this:

    private void DeployGrabbedEvent(Camera cam, ref ZEDMat zedmat, VIEW view, ZEDMat.MAT_TYPE mattype, ref Mat cvMat, ImageUpdatedEvent updateevent)
    {
        if(zedmat == null)
        {
            zedmat = new ZEDMat((uint)zedCam.ImageWidth, (uint)zedCam.ImageHeight, mattype);
        }

        if(cvMat == null)
        {
            cvMat = SLMat2CVMat(zedmat, mattype);
        }

        ERROR_CODE err = zedManager.zedCamera.RetrieveImage(zedmat, view, ZEDMat.MEM.MEM_CPU, zedmat.GetResolution());

        if (err == ERROR_CODE.SUCCESS)
        {
            Utils.copyToMat(zedmat.GetPtr(), cvMat);
            updateevent.Invoke(cam, camMat, cvMat);
        }
    }

Note that I pass the ZEDMat created pretty much the normal way (but using CPU memory) and an OpenCV mat I created with the SLMat2CVMat function. updateevent sends out that cvMat with the data copied, along with the in-game UnityEngine.Camera reference, and the OpenCV Mat that describes the camera's parameters, but you can pass whatever you'd like.

We actually have a ZED+Unity+OpenCV sample in code review right now, with a companion blog, so hopefully we'll have a full working sample for you soon.

Cool, I gave your suggestions a try and everything seems to be working well with a dummy AR camera target. I'm currently trying to figure out how to get the AR objects now into view of the left zed camera but seem to having some trouble with it, probably since I based my code off an example using a static webcam. Ultimately I am hoping to use these markers to provide reference points for the ZED but that is a few steps further along in the plan. If possible I would love to take a look at the ZED/OpenCV sample you guys are working on, code review or not. I'm also happy to provide feedback on it if I can figure out how to improve on anything. Please let me know if this is possible, I would love get this position syncing stuff working as soon as I can so I can really let loose with these ZED minis in multiplayer.

Ah, that next part is complicated. I think it's much better if I just send you an early version of the sample. :)

Can you email me at chris.orris@stereolabs.com?

FYI if you're using ArUco markers, that's what the package is designed for and you should be able to inherit from some included classes to make things easier. But if not, the main class that handles the camera images should still be helpful.

Sounds great. I'm closing this issue ticket now since my initial trouble with converting ZEDMat's to OpenCv Mats was solved by your earlier suggestions.

stereolabs / zed-unity

ZEDMat to OpenCvMat #73