While using the OpenCVSharp4 wasm runtime Cv2.ImDecode and Mat.FromImageBytes fail for jpg and png images, while bmp images work.

[LostBeard]

Summary of your issue

While using the OpenCVSharp4 wasm runtime Cv2.ImDecode and Mat.FromImageBytes fail for jpg and png images, while bmp images work.

Environment

Blazor wasm using OpenCVSharp4 and the OpenCVSharp4 wasm runtime

What did you do when you faced the problem?

I cloned your opencvsharp_blazor_sample repo and it loads the bitmap image "images/Mandrill.bmp" correctly. I then modified it to load a png and then jpg and both of those fail. The resulting Mats are empty.

I modified OpenCVSharpSample.razor method OnAfterRenderAsync with the below code:

Example code:

        // bmp
        var srcBytes = await httpClient.GetByteArrayAsync("images/Mandrill.bmp");
        srcMat ??= Mat.FromImageData(srcBytes);
        Console.WriteLine($"srcBytes.length == {srcBytes.Length} srcMat.Empty() == {srcMat.Empty()}");
        // jpg
        var jpgBytes = await httpClient.GetByteArrayAsync("images/test-image-1.jpg");
        var jpgMat = Mat.FromImageData(jpgBytes);
        Console.WriteLine($"jpgBytes.length == {jpgBytes.Length} jpgMat.Empty() == {jpgMat.Empty()}");
        // png
        var pngBytes = await httpClient.GetByteArrayAsync("images/test-image-2.png");
        var pngMat = Mat.FromImageData(pngBytes);
        Console.WriteLine($"pngBytes.length == {pngBytes.Length} pngMat.Empty() == {pngMat.Empty()}");

Output:

srcBytes.length == 196662 srcMat.Empty() == False
jpgBytes.length == 250239 jpgMat.Empty() == True
pngBytes.length == 3298424 pngMat.Empty() == True

What did you intend to be?

srcBytes.length == 196662 srcMat.Empty() == False jpgBytes.length == 250239 jpgMat.Empty() == False pngBytes.length == 3298424 pngMat.Empty() == False

Workaround:

Current workaround, in the browser, is to use an Image element to load the image, then draw it onto a canvas, read the canvas imageData, create a Mat of appropriate size and of the format CV_8UC4, and then write the image RGBA bytes to the Mat using Marshal.Copy. Quite a bit more effort than a single call but it works.

[LostBeard]

I think I have found the issue (not a bug.) It looks like png and jpeg support is not enabled for the wasm runtime build.

https://github.com/shimat/opencvsharp/blob/f3ff42260aff5d6257d1d2a6aefca3954f071d7c/.github/workflows/wasm.yml#L124-L126

Are these disabled because they do not work in the wasm runtime?

I want to do a lot of Blazor wasm demo projects with OpenCVSharp using the wasm runtime in Blazor web assembly. I wrote an open source project BlazorJS that adds multithreading to Blazor wasm and I want to put it to use with OpenCVSharp. I may try to setup a build environment for building the OpenCVSharp4 wasm runtime and testing some of the disabled features. If you can point me in the right direction it would really help.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.