[Feature request] Return a numpy Ndarray from Mat

[simonbuehler]

When using your wonderful library with keras.net there is conversion from opencv mat to numpy ndArray missing. There is an implementation ready at SharpCV mat code and it would be awesome if NumSharp could be incooperated to feed opencv data to a keras Model.predict() method

[shimat]

Thanks for useful information. However, in C#, I don't use numpy(NDArray) much, so I'll probably not be able to maintain it. I think it is better to use OpenCvSharp and SharpCV together. SharpCV uses OpenCvSharp's native bindings.

[simonbuehler]

fair enough, i found the proper way to do it , so no need to implement:

using Numpy;

Mat frameMat = capture.RetrieveMat();
var graymat = frameMat.CvtColor(ColorConversionCodes.BGR2GRAY);
graymat.GetArray(out byte[] plainArray); //there it is, c# array for nparray constructor
NDarray nDarray = np.array(plainArray, dtype: np.uint8); //party party

[simonbuehler]

the problem with SharpCv is it implements only a very small subset of opencv functionaility and i haven't found a way for casting e.g. from opencvsharps mat to Sharpcv Mat

[shimat]

You may be able to do that using the Mat(IntPtr) constructor. (I haven't tried it) https://github.com/SciSharp/SharpCV/blob/master/src/SharpCV/Core/Mat.cs#L118

var mat1 = new OpenCvSharp.Mat("foo.jpg") { 
    IsEnabledDispose = false // prevent multiple deletings of native cv::Mat* pointer
}; 

var mat2 = SharpCV.Mat(mat1.CvPtr);

...

mat2.Dispose();
// mat1.Dispose does not need to be called

[ss04661]

It seems to work，but may be leak memory `

public static class Extension
{
    public static Mat ToU8(this Mat mat)
    {
        if (mat.Type().Depth == MatType.CV_8U)
        {
            return mat;
        }
        else
        {
            var temp = new Mat();
            mat.ConvertTo(temp, MatType.CV_8U);
            return temp;
        }
    }
    public static unsafe NDArray GetNDArray(this Mat mat)
    {
        // we pass donothing as it keeps reference to src preventing its disposal by GC
        //mat.IsEnabledDispose = false;
        var _matType = mat.Type();
        Shape shape = new Shape(mat.Height, mat.Width, mat.Channels());
        switch (_matType.Depth)
        {
            case MatType.CV_8U:
                {
                    var block = new UnmanagedMemoryBlock<byte>(mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<byte>(block), shape);
                    return new NDArray(storage);
                }
            case MatType.CV_8S:
                {
                    var block = new UnmanagedMemoryBlock<sbyte>((sbyte*)mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<sbyte>(block), shape);
                    return new NDArray(storage);
                }
            case MatType.CV_16U:
                {
                    var block = new UnmanagedMemoryBlock<ushort>((ushort*)mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<ushort>(block), shape);
                    return new NDArray(storage);
                }
            case MatType.CV_16S:
                {
                    var block = new UnmanagedMemoryBlock<short>((short*)mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<short>(block), shape);
                    return new NDArray(storage);
                }
            case MatType.CV_32S:
                {
                    var block = new UnmanagedMemoryBlock<Int32>((Int32*)mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<Int32>(block), shape);
                    return new NDArray(storage);
                }
            case MatType.CV_32F:
                {
                    var block = new UnmanagedMemoryBlock<float>((float*)mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<float>(block), shape);
                    return new NDArray(storage);
                }
            case MatType.CV_64F:
                {
                    var block = new UnmanagedMemoryBlock<double>((double*)mat.DataPointer, shape.Size, () => DoNothing(mat));
                    var storage = new UnmanagedStorage(new ArraySlice<double>(block), shape);
                    return new NDArray(storage);
                }
            default:
                throw new NotImplementedException($"Can't find type: {_matType}");
        }
    }

    public static unsafe Mat GetMat(this NDArray narray)
    {
        var dtype = narray.dtype;
        int depth;

        if (dtype == typeof(byte))
        {
            depth = MatType.CV_8U;
        }
        else if (dtype == typeof(sbyte))
        {
            depth = MatType.CV_8S;
        }
        else if (dtype == typeof(short))
        {
            depth = MatType.CV_16S;
        }
        else if (dtype == typeof(ushort))
        {
            depth = MatType.CV_16U;
        }
        else if (dtype == typeof(Int32))
        {
            depth = MatType.CV_32S;
        }
        else if (dtype == typeof(float))
        {
            depth = MatType.CV_32F;
        }
        else if (dtype == typeof(double))
        {
            depth = MatType.CV_64F;
        }
        else
        {
            throw new NotImplementedException($"not support datatype: {dtype}");
        }
        if (narray.shape.Length != 3)
        {
            throw new NotImplementedException($"not support shape.Length: {narray.shape.Length}");
        }
        int row = narray.shape[0];
        int cols = narray.shape[1];
        int channels = narray.shape[2];
        var _mattype = MatType.MakeType(depth, channels);
        return new NaMap(row, cols, _mattype, narray);
    }

    [MethodImpl(MethodImplOptions.NoOptimization)]
    private static void DoNothing(Mat ptr)
    {
        var p = ptr;
    }
}

public unsafe class NaMap:Mat
{
    private readonly NDArray narray;
    public NaMap(int rows, int cols, MatType type, NDArray narray, long step = 0):
        base(rows, cols, type, (IntPtr)narray.Unsafe.Address, step)
    {
        this.narray = narray;
    }
}

`

[simonbuehler]

nice, but where do you suspect memleak?

[yazeed44]

Hello @simonbuehler @shimat mat.GetArray(out byte[] plainArray);

The above results in an exception OpenCvSharp.OpenCvSharpException: 'Mat data type is not compatible: CV_8UC4'

Any idea on how to fix it?

[simonbuehler]

How is your mat filled?

[yazeed44]

I'm getting my video feed as a BitmapSource, and then convert it to mat using BitmapSourceConverter

CascadeClassifier faceCascade = new CascadeClassifier("haarcascade_frontalface_default.xml");
// colorImage is a BitmapSource
// 1280 x 720
using (var src = BitmapSourceConverter.ToMat(colorImage))
{
int size = 4;
Mat im = src.Flip(FlipMode.XY);
Mat mini = im.Resize(new Size((int)im.Size().Height / size, (int) im.Size().Width / size));
Rect[] faces = faceCascade.DetectMultiScale(mini);

foreach (Rect faceRect in faces)
{
    int x = faceRect.X * size, y = faceRect.Y * size, w = faceRect.Width * size, h = faceRect.Height * size;
    Mat faceImg = im[new Rect(x, y, w, h)];
    Mat resized = faceImg.Resize(new Size(224, 224));
    Mat normalized = resized / 255.0;
    Mat reshaped = normalized.Reshape(0, new int[] { 1, 224, 224, 3 });
    // Need to convert to NDarray to plug into keras model prediction
    NDarray reshapedNdArray = np.vstack(np.array(MatToNDArray(reshaped)));

[simonbuehler]

you have reshaped to four channels and the conversion only supports three, you have to fix which part of the matrix should be used

[hopeko]

Try this.

Mat to NDArray

public static NDArray ToNDArray(this Mat mat)
{
    var matType = mat.Type();
    var channels = mat.Channels();
    var size = mat.Rows * mat.Cols * channels;
    var shape = channels == 1 ? new Shape(mat.Rows, mat.Cols) : new Shape(mat.Rows, mat.Cols, channels);
    if (matType == MatType.CV_32SC1 || matType == MatType.CV_32SC2)
    {
        var managedArray = new int[size];
        Marshal.Copy(mat.Data, managedArray, 0, size);
        var aslice = ArraySlice.FromArray(managedArray);
        return new NDArray(aslice, shape);
    }
    if (matType == MatType.CV_32FC1)
    {
        var managedArray = new float[size];
        Marshal.Copy(mat.Data, managedArray, 0, size);
        var aslice = ArraySlice.FromArray(managedArray);
        return new NDArray(aslice, shape);
    }
    if (matType == MatType.CV_64FC1)
    {
        var managedArray = new double[size];
        Marshal.Copy(mat.Data, managedArray, 0, size);
        var aslice = ArraySlice.FromArray(managedArray);
        return new NDArray(aslice, shape);
    }
    if (matType == MatType.CV_8UC1 || matType == MatType.CV_8UC3 || matType == MatType.CV_8UC4)
    {
        var managedArray = new byte[size];
        Marshal.Copy(mat.Data, managedArray, 0, size);
        var aslice = ArraySlice.FromArray(managedArray);
        return new NDArray(aslice, shape);
    }

    throw new Exception($"mat data type = {matType} is not supported");
}

public static unsafe NDArray ToNDArrayUnsafe(this Mat mat)
{
    var matType = mat.Type();
    var channels = mat.Channels();
    var size = mat.Rows * mat.Cols * channels;
    var shape = channels == 1 ? new Shape(mat.Rows, mat.Cols) : new Shape(mat.Rows, mat.Cols, channels);
    if (matType == MatType.CV_32SC1 || matType == MatType.CV_32SC2)
    {
        int* ptr = (int*)mat.DataPointer;
        var block = new UnmanagedMemoryBlock<int>(ptr, shape.Size, () => DoNothing(IntPtr.Zero));
        var storage = new UnmanagedStorage(new ArraySlice<int>(block), shape);
        return new NDArray(storage);
    }
    if (matType == MatType.CV_32FC1)
    {
        float* ptr = (float*)mat.DataPointer;
        var block = new UnmanagedMemoryBlock<float>(ptr, shape.Size, () => DoNothing(IntPtr.Zero));
        var storage = new UnmanagedStorage(new ArraySlice<float>(block), shape);
        return new NDArray(storage);
    }
    if (matType == MatType.CV_64FC1)
    {
        double* ptr = (double*)mat.DataPointer;
        var block = new UnmanagedMemoryBlock<double>(ptr, shape.Size, () => DoNothing(IntPtr.Zero));
        var storage = new UnmanagedStorage(new ArraySlice<double>(block), shape);
        return new NDArray(storage);
    }
    if (matType == MatType.CV_8UC1 || matType == MatType.CV_8UC3 || matType == MatType.CV_8UC4)
    {
        byte* ptr = (byte*)mat.DataPointer;
        var block = new UnmanagedMemoryBlock<byte>(ptr, shape.Size, () => DoNothing(IntPtr.Zero));
        var storage = new UnmanagedStorage(new ArraySlice<byte>(block), shape);
        return new NDArray(storage);
    }

    throw new Exception($"mat data type = {matType} is not supported");
}

[MethodImpl(MethodImplOptions.NoOptimization)]
private static void DoNothing(IntPtr ptr)
{
    var p = ptr;
}

NDArray to Mat

public static Mat ToMat(this NDArray nDArray) => 
    new Mat(nDArray.shape[0], nDArray.shape[1], nDArray.GetMatType(), (Array)nDArray);

public unsafe static Mat ToMatUnsafe(this NDArray nDArray) =>
    new Mat(nDArray.shape[0], nDArray.shape[1], nDArray.GetMatType(), new IntPtr(nDArray.Unsafe.Address));

public static MatType GetMatType(this NDArray nDArray)
{
    int channels = nDArray.ndim == 3 ? nDArray.shape[2] : 1;
    return nDArray.typecode switch
    {
            NPTypeCode.Int32 => channels == 1 ? MatType.CV_32SC1 : 
            channels == 2 ? MatType.CV_32SC2 :
            throw new ArgumentException($"nDArray data type = {nDArray.typecode} & channels = {channels} is not supported"),

            NPTypeCode.Float => channels == 1 ? MatType.CV_32FC1 :
            throw new ArgumentException($"nDArray data type = {nDArray.typecode} & channels = {channels} is not supported"),

            NPTypeCode.Double => channels == 1 ? MatType.CV_64FC1 :
            throw new ArgumentException($"nDArray data type = {nDArray.typecode} & channels = {channels} is not supported"),

            NPTypeCode.Byte => channels == 1 ? MatType.CV_8UC1 :
            channels == 3 ? MatType.CV_8UC3 :
            channels == 4 ? MatType.CV_8UC4 :
            throw new ArgumentException($"nDArray data type = {nDArray.typecode} & channels = {channels} is not supported"),

            _ => throw new ArgumentException($"nDArray data type = {nDArray.typecode} is not supported")
    };
}

[moon341]

What is Arrayslice in your code in this class "public static NDArray ToNDArray(this Mat mat)"?

[NarwhalRoger]

@simonbuehler Hi. I am currently also working with Keras.NET and got seriously stuck on model.predict method. Keras accepts only pure Numpy NDarrays (even scisharp's numsharp NDarrays don't work), and I can't find a way to convert my Mat objects (images) to Numpy arrays. I tried your method described above, but my model accepts only 3 channeled inputs, while you are using grayscale which I believe is 1 channel. Can you explain how did you make it work? Did you make model that accepts single channeled input? Or how did you convert images to numpy arrays?

I tried converting OpenCvSharp.Mat object to numpy NDarray but that only works on 1 channel Mat objects. I also tried using Bitmap, but there is no clear Bitmap to numpy Ndarray conversion.

[simonbuehler]

its been a while and i'm not into it atm, maybe this helps you

using (var graymat = frameMat.Resize(Size.Zero, scaling, scaling, InterpolationFlags.Nearest).CvtColor(ColorConversionCodes.BGR2GRAY))
            {
                Rect[] rects = cascadeClassifier
                    .DetectMultiScale(graymat, 1.1, 5, HaarDetectionType.ScaleImage, new Size(30, 30))
                    .OrderByDescending(b => b.Width * b.Height)
                    .Take(10)
                    .ToArray();
                for (int i = 0; i < rects.Length; i++)
                {
                    using (Py.GIL())
                    {
                        Mat roi = graymat[rects[i]];
                        Cv2.Resize(roi, roi, new Size(48, 48));
                        roi.GetArray(out byte[] plainArray);
                        NDarray floatND = np.array(plainArray, dtype: np.uint8);
                        floatND = floatND.reshape(48, 48).astype(np.float_) / 255;
                        NDarray final = ImageUtil.ImageToArray(floatND);
                        final = np.expand_dims(final, 0);
                        NDarray res = model.Predict(final);
                        Results.Emotions = Results.Emotions is null ? res : np.concatenate(new[] { Results.Emotions, res }, axis: 0);
                    }
                    rects[i] = new Rect(rects[i].Location * (1 / scaling), new Size(rects[i].Width * (1 / scaling), rects[i].Height * (1 / scaling)));
                }
                Results.Rects = rects;
            }

i think you will have to do some array reshape to stuff it into yout model.predict() with matching dimensions / structure.

[NarwhalRoger]

@simonbuehler Thanks I will try this, but can I ask about ImageUtil.ImageToArray function? I tried it before, but it gave me this unusuall exception: "System.NotImplementedException:'Type is not yet supported: Mat. Add it to 'ToPythonConversions''.

This exception made me think that this function is not implemented yet, but looks like it works in your code

[simonbuehler]

it operating on NDarray floatND not Mat

[NarwhalRoger]

And could you please tell me if your model was accepting 3-channeled inputs or single-channeled inputs?