Using SceneCaptureVideoSource on Android running ARCore

[astaikos316]

Describe the bug Currently have a test scene set up, where I am trying to source a video session from an Android Tablet to a Windows PC. In the application I am using ARCore for an Augmented Reality experience. Upon running the test application on the tablet, it errors out because the SceneCaptureVIdeoSource I am trying to use to capture frames from an In-Game camera I have looking at a RenderTexture generated by the ARCamera in order to see the video and 3D AR objects is dependent on AsyncGPUReadback which is not supported by OpenGLES3. Vulcan cannot be used since ARCore does not support it.

To Reproduce Set up SceneCaptureVideo scene with ARFoundation components. Build and run on Android device. AsyncGPUReadback error shown when trying to run.

Expected behavior Be able to send scene camera frames to remote peer. Are there any ways available to accomplish what I am trying to send currently?

Environment

• Platform: Android Pie Tablet to Windows Unity Editor PC Build
• Architecture: Android - ARM64. PC x64
• Unity version: 2019.3.9f1
• Target device: Android Pie to Windows PC Build

[djee-ms]

Async GPU readback is not supported with OpenGL. To the best of my knowledge there is no workaround, especially as OpenGL is being deprecated as I understand so will never see support for it, and Unity encourages developers to move to Vulkan. If that cannot be used, then we're in an impasse, as we cannot access the camera rendered images. So there is no way for us to implement that feature.

[astaikos316]

@djee-ms I have managed to get something functioning where I am able to send the AR enabled video from Android tablet to Windows using some concepts that were a precursor to the Async GPU readback api. However the problem I am seeing is that on the local source android side the video looks smooth and good. On the windows receiver, the frame rate of the video from the android tablet is awful. Not sure where to look to help improve performance. Any help would be greatly appreciated.

[djee-ms]

• Are you using a Release build of everything?
• What's the actual (measured) framerate on the remote?
• What's the video framerate of the camera? Does it have adaptive framerate (generally does)? What's the (measured) difference between that and the playback framerate on the remote?
• What's the CPU load on both sides?

[astaikos316]

I am running in the editor on the windows side. I have android built to debug actually because for some reason even building libwebrtc to release, android studio was complaining about not finding debug directory. I don't have a profiler running on the tablet at the moment so I cannot see CPU usage. Received framerate I see in editor is about 1.0 and below.

[astaikos316]

Just ran a profiler for the tablet, it’s running between 50-60% cpu load and about 800MB memory.

[astaikos316]

@djee-ms I've been using the UniformColorVideoSource script as an example. When I set the frame width and frame height to 10% of what they should be, I get the expected framerate at the destination. Desired frame size is (2560x1600). Can only get good performance at (256x160) which really clips the camera image.

[djee-ms]

What video codec are you using? Are you expecting an Android phone/tablet to have enough horsepower to encode in real time at 30fps a 2560x1600 video on CPU (there's no hardware acceleration that I know of on Android)? Because from experience I am a bit skeptical that this will happen, at least not for very long until the battery is dead, and I also doubt you will have the bandwidth to stream that much (unless you only target local network). I don't know what your application is but I'll do the math first on checking that this resolution/framerate/bandwidth is acceptable, and try a more reasonable resolution too (e.g. something around 720p which is what we use on HoloLens, which is also a mobile device).

And use a Release build, because any kind of performance issue in Debug is not something we expect to investigate, libwebrtc is well known to be slow in Debug and I've seen it running at 1-2fps between two beefy developer machines with hardware accelerated encoding. If there's an issue building that then let's investigate that one first.

[astaikos316]

The latest pull of master I took removed the ability to select codecs and I haven't noticed what is negotiated in the SDP messages at the moment. I've tried downsampling the resolution but it has an odd effect on the tablet where it looks like it is zooming in the camera instead of having same camera view with lower resolution. And yes the latest pull did not allow me to build release in android studio. During build it said it was looking for debug folder for libwebrtc. Could not find a reason why.

[djee-ms]

The latest pull of master I took removed the ability to select codecs

What do you mean? The only thing that doesn't work since the new transceiver API on master is per-track (or rather, per-transceiver) codec selection, which is more fine grained but has more limited use and was never added in the first place because it's fairly involved. Instead the original per-peer connection codec selection was retained; we are using it regularly to test the H.264 encoder (the Google implementation defaults to VP8 otherwise).

I've tried downsampling the resolution

You should select a lower capture resolution, not downsample anything. Selecting a lower resolution means less work to capture and encode, downsampling means more work to still capture at high resolution then process the frame to reduce its resolution. See DeviceVideoTrackSource.GetCaptureFormatsAsync() to list available formats, and the fields in LocalVideoDeviceInitConfig to select one.

And yes the latest pull did not allow me to build release in android studio. During build it said it was looking for debug folder for libwebrtc. Could not find a reason why.

In ran our internal release pipeline several times in the past few days, which builds the Release variant of the Android library (see this yaml pipeline), and we have no issue. Can you please open an issue and provide more details? Which part is looking for a debug folder, the libwebrtc build from Google (built with build.sh) or the mrwebrtc.aar from MixedReality-WebRTC (built with the Android Studio project / Gradle)?

[astaikos316]

I do not see any option in the PeerConnection Inspector window in unity to select codec. I will take a look at the capture resolution information you provided. The Android Studion build project fir mrwebrtc.aar errors out on build for me looking for debug folder containing libwebrtc.

[astaikos316]

@djee-ms I’ve taken a step back and started trying to tackle this using a different API. ARFoundation has a class called XRCpuImages. I believe this should help give similar functionality but when I try to pass the data to an ARGB32frame, it does not want to work. I know the byte pointers are valid and the data is there because I am also putting it on a Texture2D to verify. Could definitely use the help of the community as this would provide great Mixed Reality functionality for other devices.

[djee-ms]

when I try to pass the data to an ARGB32frame, it does not want to work.

What does "it does not want to work" mean? We need more information than that to help. To which method are you passing the frame? What is the frame resolution? Are you using the correct encoding (0xAARRGGBB little endian, so (0xBB,0xGG,0xRR,0xAA) in byte memory order)? Can you confirm the video connection is established, that is some video callback is called on the receiver? What is the content of the data on that receiving side? Is the received resolution correct? etc.

[astaikos316]

I am trying to confirm that the byte data i am pulling from XRCPUImage Async conversion with ARFoundation, which I believe should give similar performance and capability. Frame resolution i am trying is 640x480 to start with. I believe i am in little endian encoding. I havent even tried getting video connection established because I am first just testing that i get a local video render which I am not.

[Kerugal]

@astaikos316 Have you made any progress? Because I have the same problem and want to combine WebRTC with AR Foundation.

[astaikos316]

I have not been able to get the frames from XCPUImage to an ARGBFrame. Have you tried the same method @Kerugal ?

[Kerugal]

@astaikos316 I tried to convert it to an ARGB frame, but I can't get it to work. Another approach to my problem is the implementation of a WebView in which a web application runs with WebRTC, from which I grab the texture byte array of the AR camera.

I would be very grateful for any further suggestions!

[dhaminitinAltran]

Hi, I am able to start the call but unable to send the camera feed were you able to start the camera feed as well?