search

EnoxSoftware / OpenCVForUnity

OpenCV for Unity (Untiy Asset Plugin)
https://assetstore.unity.com/packages/tools/integration/opencv-for-unity-21088
557 stars 175 forks source link

Tearing on Mat in WebCamTextureToMatExample on Android #161

Closed JoeyVrowl closed 1 year ago

JoeyVrowl commented 1 year ago

When running the WebCamTextureToMatExample scene, we get image tearing on Android. It happens on both the front and rear cameras. I suspect it happens when the app framerate is lower than the camera, because using a slower phone or artificially lowering performance with Thread.Sleep() significantly increases tearing.

Specifically, it seems like there are 2 frames overlaid onto each other in a strip-like fashion. There is no pattern, the "cutlines" are in different places every frame, and there is a different amount of them depending on the performance. The cutlines are not always continuous, as you can see two 1-pixel steps on the F12 key on the keyboard.

The issue does not occur when directly showing the Unity WebCamTexture. The offending code is unfortunately in the prebuilt library file. I have figured this because when the webcam example calls Utils.webCamTextureToMat(webCamTexture, rgbaMat, colors); in Update(), I can save the rgbaMat using imwrite, resulting in the example image posted below (with R and B channels swapped because I was lazy). This leads to Utils.OpenCVForUnity_TextureToMat, which is end of the line for me since that is in the prebuilt library.

It was tested on several phones (Galaxy S22, S8, S7). It does not happen in the Editor (Windows), and not on iOS.

tearing

EnoxSoftware commented 1 year ago

Thank you for your reporting.

Could you tell me the environment you tested? OpenCVForUnity version :
Unity version : Editor platform :

Is there a code to reproduce this problem?

JoeyVrowl commented 1 year ago

OpenCVForUnity version : 2.4.9 (also tried 2.5.1 in a fresh project) Unity version : 2021.3.4f1 and 2021.3.16f1 (latest LTS) Editor platform : Windows

As I mentioned, it happens in your WebCamTextureToMatExample scene. I investigated a little further, and it seems the webCamTexture.GetPixels32() call in Utils.webCamTextureToMat returns the corrupt data. This can be tested by making another Texture2D, and using SetPixels32() on it with the Color32 data from the webcam texture. The new texture also shows the tearing, without it having been converted to a Mat. A fresh project with just OpenCV also exhibits the issue, and so does a fresh project without OpenCV and just the SetPixels32 test.

After this, I tried disabling Vulkan as a graphics API, and it fixed the problem. When searching for Vulkan and WebCamTexture in Unity, it became clear that Vulkan has a history of causing problems [1] [2]