search

Unity-Technologies / arfoundation-samples

Example content for Unity projects based on AR Foundation
Other
3.07k stars 1.15k forks source link

How do I get the native CVPixelBuffer? #109

Closed davidvjc closed 5 years ago

davidvjc commented 5 years ago

Hey - I can't find a reference to the iOS CVPixelBuffer in ARFoundation.

In ARKit you could find it in the ARSessionNative.

Any ideas?

tdmowrer commented 5 years ago

See https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@1.0/manual/cpu-camera-image.html

davidvjc commented 5 years ago

Thanks for the link, that's what I'm looking for!

So the texture/camera image I get from this seemed to be rotated 90deg. I imagine that's because the camera output is in Landscape.

Is there a way to rotate it in Unity using the above tools?

tdmowrer commented 5 years ago

No, there is no method to rotate the image. This is the same image you get from the CVPixelBuffer, i.e., unrotated.

There are some available transformations you can apply, like mirroring (flipping across the X or Y axis). However, the assumption is that you are passing the image data off to a computer vision processing library (e.g, OpenCV). Your image processing library can presumably handle different orientations (or has functionality to rotate the image if necessary), so this was not deemed necessary.

Is our assumption correct, or do you require some rotation functionality built into the camera image API?

vincentfretin commented 5 years ago

@davidvjc I think you need to know what is the device orientation and do some transforms yourself. https://docs.unity3d.com/ScriptReference/Input-deviceOrientation.html may help. Mozilla did some experiments with opencv in javascript and ARKit in ObjectiveC, and they did some transforms based on device orientation to get the aruco marker position. Relevant parts of the code may interest you: https://github.com/mozilla-mobile/webxr-ios/blob/7b1649d9dabef91e4aa826fdefd1493ec8dae23a/XRViewer/Utils/Utils.m#L12 https://github.com/mozilla-mobile/webxr-ios/blob/4b5889e5fb2ad75c422da44c5860c6effdd84fa7/XRViewer/Resources/webxr.js#L6089 https://github.com/mozilla/webxr-polyfill/blob/f6b85d656260398c4e2f78b73349d9ae567486ae/examples/opencv-aruco/worker.js#L279 Of course you need to transpose this to Unity. I have zero experience in Unity, but I plan to write an app with ARFoundation and opencv in the coming months.

davidvjc commented 5 years ago

@vincentfretin Thanks! Very helpful. Hope it goes well.

@tdmowrer In this case I have to rotate it manually. I do think it would be useful to be able to rotate directly in ARFoundation. If you can flip it, why not rotate it? My 2 cents... :)

tdmowrer commented 5 years ago

@davidvjc

In this case I have to rotate it manually.

Can you share what you are doing with the image? What is the use case?

vincentfretin commented 5 years ago

@davidvjc What's your use case? Will you use opencv or not? It's not clear in your comments. Recent discussion on this issue may be interesting to you as well: https://github.com/google-ar/arcore-unity-sdk/issues/527#issuecomment-479473904

pzoltowski commented 5 years ago

some example how to integrate it with opencv would be very helpful, e.g. just applying sobel or canny edge detection in c++ using opencv. Is there also a fine grained way to choose video stream format? e.g. 3d depth and/or frame rate?

manwithsteelnerves commented 2 years ago

@tdmowrer Can you please give a hint on converting XRCpuImage to CVPixelBuffer?

tdmowrer commented 2 years ago

@tdmowrer Can you please give a hint on converting XRCpuImage to CVPixelBuffer?

AR Foundation does not provide a way to get the CVPixelBuffer pointer, but you can get the raw data for nearly free (it's just a memcpy) using the XRCpuImage. If you actually need a pointer to the CVPixelBuffer, you'd have to access it through the ARFrame, which you can get from one of the native pointers (look for "extensions" in the docs). Which do you need?

manwithsteelnerves commented 2 years ago

Well! Actually I want CMSampleBuffer for using it in my other libraries on native. For that I see ARFrame capturedImage is a wrapper for XRCpuImage and it returns CVPixelBuffer. I can convert CVPixelBuffer to an UIImage or CMSampleBuffer for my needs.

I checked the extensions doc but there is no much info. If ARFoundation platform implementation is open source it might have helped a lot ;)

On Android, I yet need to explore a way to get native pixel buffer.

So, Only chance I have now is to go with a memcpy through convert methods and pass it to my other code? It would be great if I can avoid a memcpy too as I need to do it almost every frame which may not be worth it if its going to be free on native.

Thanks @tdmowrer for looking into it. Really appreciate it!

tdmowrer commented 2 years ago

Well, if you want the raw memory, you don't need to convert it. Just GetPlane on the XRCpuImage to get each image plane: https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.2/api/UnityEngine.XR.ARSubsystems.XRCpuImage.Plane.html

Note this is very platform specific. See https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.2/api/UnityEngine.XR.ARSubsystems.XRCpuImage.Format.html for details.

manwithsteelnerves commented 2 years ago

Well, if you want the raw memory, you don't need to convert it. Just GetPlane on the XRCpuImage to get each image plane: https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.2/api/UnityEngine.XR.ARSubsystems.XRCpuImage.Plane.html

Note this is very platform specific. See https://docs.unity3d.com/Packages/com.unity.xr.arsubsystems@4.2/api/UnityEngine.XR.ARSubsystems.XRCpuImage.Format.html for details.

Thanks for the details. Can I get each plane native array data and merge them for cvpixelbuffer?

tdmowrer commented 2 years ago

If I remember correctly, each plane is the result of CVPixelBufferGetBaseAddressOfPlane.

The XRCpuImage will give you the raw data of each plane, but if you really need a CVPixelBuffer object, I suggest you use the extension mechanism I mentioned earlier to get that pointer rather than trying to reconstruct it from raw data. Someone asked a similar question in https://github.com/Unity-Technologies/arfoundation-samples/issues/517 and has some Objective-C code to get you started. Once you've got the session's currentFrame, you can get the CVPixelBuffer from its capturedImage property.

manwithsteelnerves commented 2 years ago

I see. Thanks for the reference. Looks like extending path is the way to go as i can get the direct access to the buffer. Just a last one. How the same can be done on Android? Is there a documentation on how to extend on Android (for ARCore)? I need a native bytebuffer (pixel data) for passing it to my libraries.

tdmowrer commented 2 years ago

You can use a similar trick to get a pointer to ARCore's ArFrame, then you can get whatever you need using the C API. Note that you'll need the ArFrame instead of the ArSession, which I'm pretty sure is the pointer you get from XRCameraSubsystem.TryGetLatestFrame in the ARCamera.frameReceived callback.

However, if all you need is a bytebuffer, then you should just use the XRCpuImage plane. Depends on whether you need the native ArImage object or just the bytes. In ARCore's case, the ArImage is a bit simpler than iOS's CVPixelBuffer.

manwithsteelnerves commented 2 years ago

You can use a similar trick to get a pointer to ARCore's ArFrame, then you can get whatever you need using the C API. Note that you'll need the ArFrame instead of the ArSession, which I'm pretty sure is the pointer you get from XRCameraSubsystem.TryGetLatestFrame in the ARCamera.frameReceived callback.

However, if all you need is a bytebuffer, then you should just use the XRCpuImage plane. Depends on whether you need the native ArImage object or just the bytes. In ARCore's case, the ArImage is a bit simpler than iOS's CVPixelBuffer.

Actually ARFrame ptr will work for me as well. I don't see any access to it actually as XRCpuImage.api is totally hiding the native pointers. Anything I'm missing?

tdmowrer commented 2 years ago

Right, it isn't part of the XRCpuImage; you get it from the XRCameraFrame (I think).

manwithsteelnerves commented 2 years ago

@tdmowrer thanks alot :) I will give a try!

tdmowrer commented 2 years ago

See also https://github.com/Unity-Technologies/arfoundation-samples/issues/347#issuecomment-635478726

manwithsteelnerves commented 2 years ago

@tdmowrer Finally got access to XRCameraFrame on C# end but stuck with undocumented information.

nativePtr of XRCameraFrame points to a struct with version + native void* pointer. But now, I see there is no way to use it on java end as I can't make an AndroidJavaObject instance from the available data to pass it to java code.

I see on iOS its straight forward but any hints on how to handle it on Java? If I convert struct/void* to IntPtr through Marshal methods, I see AndroidJavaObject's constructor which is taking IntPtr is not exposed! It's set to internal.

tdmowrer commented 2 years ago

It's not a Java object. You need to use the ARCore C API: https://developers.google.com/ar/reference/c

You can p/invoke these functions from C#.

manwithsteelnerves commented 2 years ago

Ah! Now it makes sense. My native libs are written in java. So first I need to access it(nativeptr) with ndk and pass it to java. Right?

tdmowrer commented 2 years ago

I don't think a pointer from the ndk can be converted to a its JavaObject equivalent, e.g. a native ArFrame to a Java ArFrame. I'm not a Java programmer though so could be wrong.

manwithsteelnerves commented 2 years ago

Java's ARCore lib is based on arcore's c Api. So there should be a way to convert from ArFrame of NDK to Frame of java's. Will keep you updated!

manwithsteelnerves commented 2 years ago

I went through that approach

  1. I noticed Java's ARCore is just using objects created through C api under it
  2. Java libs doesn't expose construction of objects with the native handles So, this made it difficult to create Session (Java object) by passing ArSession handle.

So, for now I'm constructing my own ArImage object(on Java) by fetching native ArImage from the c api (by passing ArSession and ArFrame handles - via subsystems IntPtr structs) and passing it to my java library. I see it should work now!

manwithsteelnerves commented 2 years ago

@tdmowrer I finally dropped going through Java approach as there is no way currently to utilize the native handles through java. Most of the api is not exposed. I failed to create ArImage (extends media.Image) as media.Image abstract class intentionally doesn't allow to extend in java. Most likely the media.image objects need to be created via JNI/JDK.

So, I currently ended up using xrcpuimage apis directly but I have another way with NDK+JNI approach to avoid making a buffer copy. Additionally, I have another conversion too - Making NV21 byte buffer from YUV_420_888 format. This makes it clear path that NDK/JNI is the best approach by interfacing with native c api to avoid any kind of memory transfers.

Currently its taking nearly 16ms for the receiving the cpu image from unity to java which is most likely due to intermediate buffer creations(NativeArray -> java's nio ByteBuffer).