What's the recommended way to capture camera image in 1.7.0?

[jivanicky]

Hey, I have just been reading #221, as I am trying to use Texture Reader to read the texture using v1.7.0, and I have noticed that the ComputerVision class has been modified so that it no longer makes use of TextureReader - also it seems that when the example is running on the phone, the highres/lowres option makes no difference. Why was this changed? Is the use of TextureReader discouraged? What's the recommended way to capture camera image now?

[mpottinger]

I am also eager to know this. I think the TextureReader is being depreciated. Too bad because the CameraImageBytes YUV format is difficult for me to convert to RGB. I haven't figured it out yet.

[jivanicky]

I am trying to use OpenCV (as I am going to do some processing with the image with CV nonetheless), which should make way for a one-line conversion to RGB. However I still have issues even understanding how to get the image from ARCore to OpenCV.

Not even mentioning the fact that unlike TextureReader, CameraImageBytes still seems to return only a 640x480 image.

[mpottinger]

@jivanicky What i am doing to get it into Opencv is communicate via localhost using tcp and transferring to a Python app that way. Not sure if that would be helpful to you.

I run Linux on the phone via the UserLand app and use python Opencv in there. Not for ideal everyone though.

[mpottinger]

@jivanicky Ah yes the 640x480. Yeah nobody has explained why we are limited to that now. I am pretty sure the TextureReader is depreciated but not 100% sure.

I heard TextureReader is slow and that may be a reason. However I have not seen any explanation as to why we are now limited to 640x480. For me its not an issue because object detection like Yolo or SSD works at low res anyway.

For the YUV format i am able to get the gray-scale Y channel to display in Opencv so far. That is fairly simple, i can post the code if you like.

Converting from YUV to RGB directly in Unity would be nice. So far it looks complicated to me!

Very frustrating, it is holding up my project. Why such a complicated format.

[jivanicky]

I have finally managed to get this, altough I still seem to have a bit of problem with keeping the colors (They image has them, just incorrectly placed). Also, it is not that complicated, there is just a ton of possible issues. For me, the code works for now (apart from being grayscale and flipped, but I hope to get both fixed by tomorrow). If you want I can post it somewhere.

Also otherwise the format is not that complicated, what I perceive as a bigger problem is that I found exactly three websites where YUV420888 is specified - one of them has incorrect info, the other one is written in a such a way that it can't be understood, and the last one is VLC's website.

[jivanicky]

By the way, I believe that the real problem with 640x480 is not even the image quality, but the fixed aspect ratio. For example on my 18:9 screen, this means that only 12:9 image is going to be captured by the acquire bytes function, which also might confuse the user.

I fail to even imagine a good reason why we are not allowed to access the camera image in the same quality and aspect ratio as it is shown on the screen.

[jivanicky]

Got it, figured out the colors as well. The funny thing is, that I've studied approximately 30 stack overflow questions, forums, websites etc. to get to this point and none of them were 100% right, the final code is a mashup of them all. Will link the final code (both C# and C++ parts) first thing tommorow morning.

[mpottinger]

@jivanicky Nice, ok perfect thank you that would be awesome. I know opencv does have a conversion function, but I would have to study for a bit to figure out how to combine the channels into a numpy array/opencv mat in the corect way.

Yeah the aspect ratio thing is odd. I hope someone involved with arcore can explain that. It would make a lot more sense to match what is on the screen. Very strange.

[jivanicky]

Sorry for the wait, I only got to this today. The entire code is here: https://stackoverflow.com/questions/55495030/how-to-use-get-a-arcore-camera-image-to-opencv-in-an-unity-android-app/55495031#55495031. I have used C++ instead of Python, this also allows you to move the image data directly and very, very efficiently opposed to sending it using TCP. Feel free to ask if you have any trouble (in the SO thread, if possible).

[mpottinger]

Awesome! Can't wait to try this. I am not familiar with C++ but I probably should at least learn to use Opencv in C++. I will still have to use tcp anyway because I will be doing off-device processing, but this will do what I need.

Definitely looks similar to but slightly different from solutions I found last night looking around.

So you just append the Y, U, and V channels together sequentially in a byte array, and OpenCV does the rest.

Still no answer from anyone on why just 640x480 though :(

I also saw a Unity, not Arcore feature that may allow for reading frames at decent performance. It involves copying from the GPU asynchronously.

I haven't tried it yet but might give it a shot because i need images with the augmented objects as well.

https://docs.unity3d.com/ScriptReference/Rendering.AsyncGPUReadback.html

[mpottinger]

I figured out how to get the full resolution CPU image in Arcore 1.8. It works, I adapted it from what I saw in the computervision example. I can now get the full camera resolution with cameraimagebytes.

put this in your class variables:

private ARCoreSession.OnChooseCameraConfigurationDelegate m_OnChoseCameraConfiguration = null;

put this in Start()

m_OnChoseCameraConfiguration = _ChooseCameraConfiguration; ARSessionManager.RegisterChooseCameraConfigurationCallback(m_OnChoseCameraConfiguration); ARSessionManager.enabled = false; ARSessionManager.enabled = true;

Add this callback to the class:

private int _ChooseCameraConfiguration(List<CameraConfig> supportedConfigurations) { return supportedConfigurations.Count - 1; }

Once you add those, you should have cameraimagebytes returning the full resolution of the camera.