NumSharp.Bitmap and OpenCv

[Oceania2018]

I want to add some common methods to interoperate with OpenCv in NumSharp.Bitmap so that they can be shared. The advantage is that users can easily use NumSharp and OpenCv to process image applications.

For exampe: NdarrayToMat, MatToNdarray.

And we have to rely on this package OpenCvSharp4 which is about 1M size.

[Nucs]

static unsafe void Main(string[] args)
{
    Mat src = new Mat("olga.jpg", ImreadModes.AnyColor);
    NDArray nd = WrapWithNDArray(src); //shaped (1, src.Height, src.Width, Channels)
    Bitmap bmp = nd.ToBitmap(); //using NumSharp.Bitmap
    bmp.Save("./olga-reconstructed.jpg");
}

//this method copies Mat to a new NDArray
public static unsafe NDArray ToNDArray(Mat src)
{
    var nd = new NDArray(NPTypeCode.Byte, (1, src.Height, src.Width, src.Type().Channels), fillZeros: false);
    new UnmanagedMemoryBlock<byte>(src.DataPointer, nd.size)
        .CopyTo(nd.Unsafe.Address);

    return nd;
}

//this method wraps without copying Mat.
public static unsafe NDArray WrapWithNDArray(Mat src)
{
    Shape shape = (1, src.Height, src.Width, src.Type().Channels);
    var storage = new UnmanagedStorage(new ArraySlice<byte>(new UnmanagedMemoryBlock<byte>(src.DataPointer, shape.Size, () => Donothing(src))), shape); //we pass donothing as it keeps reference to src preventing its disposal by GC
    return new NDArray(storage);
}

[MethodImpl(MethodImplOptions.NoOptimization)]
private static void Donothing(Mat m)
{
    var a = m;
}

Similarly to these? (and backwards to Mat) (taken from https://github.com/SciSharp/NumSharp/issues/371)

[Oceania2018]

@Nucs Correct, People use OpenCv a lot. We should make NumSharp and OpenCv work better. We have combined TF.NET and NumSharp. NumSharp with OpenCv is the last piece for Computer Vision.

[AvenSun]

@Nucs @Oceania2018

It would be better we can make it more generalized (data type, image layout etc. ). as far as i know, currently there're two image layouts used by mainstream deeplearning frameworks. cuDNN (Nvidia DNN library) is much more efficient when using NCHW,
but intel GPU would prefer to use NCHW.

tensorflow  ==> NHWC (numl, height, width, channe)
pytorch ==> NCHW (num, channel, height, width)
mxnet ==> NCHW (num, channel, height, width)
caffe ==> NCHW (num, channel, height, width)

during training or inference, sometimes we might decare ndarray with data type uint8 explicitly, sometimes float32 etc.

[Nucs]

@AvenSun Our code base is deeply tied to work with tensorflow. But a simple nd.reshape should do the trick (if not, we have np.swapaxes implemented) because the data is linear. With that being, I'll also add an enum parameter allowing to choose between the libraries you mentioned that'll decide the return shape.

[AvenSun]

@Nucs yes, I knew there're several ways to transfer NCHW into NHWC and vice versa. It'll cost extra time if we don't make it at the beginning. This is more noticeable when the operation lies in loops. NumSharp should be a common numerical computing library like numpy. It should not be just for tensorflow.

[Oceania2018]

@Nucs I think NumSharp is not tied to TF.NET since v0.20. It's TF.NET tied to NumSharp.

[Nucs]

The thing is that all the common image formats provide their data in a linear way, R1G1B1R2G2B2 ... So in my opinion using NHWC is the natural way to handle images hence why I think the primary shape order should be NHWC.

Nevertheless I'll add support for NCHW