[BUG]: Frame Drop in VR Oculus Quest2 app when streaming start using Broadcast and see stream using VideoPlayer webserver template

dhruvaura commented 2 years ago

Package version

3.1.0-exp.3

Environment

* OS: Oculus Quest 2 Android
* Unity version: 2020.3.11f1
* Graphics API: OpenGLES 2
* Browser: Google Chrome (on Windows 10 DirectX12 | 8GB Ram | Intel i5 7500 | Nvidia 1070 GPU)

Steps To Reproduce

Used Broadcast to stream Oculus Quest Top camera view to browser google chrome. When streaming starts, App FPS goes down to 72 to below 35 (In Range 25-35).

Using VideoPlayer webserver template to see streaming in google chrome.
In App, Currently Two cameras available. One for Player VR camera purpose and second is for Top View camera streaming.
3D Rendering is not an issue of FPS drop. Because without streaming, app runs proper at above 72 FPS.
Streaming working proper, but main issue is app frame drop so performance of app goes down
Use Scripts " RenderStreaming.cs | Broadcast.cs | Two Camera : 1st VR Player Camera (Simple camera, no any component script added for send stream) and 2nd Camera for Top View send streaming (CameraStreamSender component added for send top view of game stream)

here some profiler analysis screenshots:

See Profiler BEFORE streaming starts:

See Profiler AFTER streaming starts:

Current Behavior

Streaming is working proper as expected, but main issue is Frame drop when streaming starts.

Expected Behavior

Expecting oculus quest 2 app good performance when start streaming. Currently vr app performance goes down, so expecting app performance good.

Anything else?

How can I make app stable FPS above 60 when streaming starts?

karasusan commented 2 years ago

@dhruvaura Thank you for sharing your issue. Could you check the performance of the app with profiler? I am not familier with the method of profiling for oculus quest 2, but found this article. https://developer.oculus.com/documentation/unity/unity-profiler-panel/

karasusan commented 2 years ago

One more question, Is OpenGLES 2 used by chrome on Oculus Quest 2?

dhruvaura commented 2 years ago

One more question, Is OpenGLES 2 used by chrome on Oculus Quest 2?

Here complete use case:

VR Game run on Oculus Quest 2 [ In Unity, I've set Graphics API to OpenGLES3, because Vulkun is not supported by oculus ]
There is simple 3D environment with two camera (One for VR Player and second is for top view camera to send streaming to webserver and see in VideoPlayer example). I've integrated broadcast render streaming feature in gameplay scene, that Top View camera is sending stream to WebApp. so I can able to see stream at /videoplayer/index.html (WebApp example). Streaming is working perfectly, but one issue is performance down when app streaming starts.
Use Scripts " RenderStreaming.cs | Broadcast.cs | Two Camera : 1st VR Player Camera (Simple camera, no any component script added for send stream) and 2nd Camera for Top View send streaming (CameraStreamSender component added for send top view of game stream)
I think Unity Render Streaming is using UDP to transfer data, so packet drop send-resend is not an issue to app performance going down.

dhruvaura commented 2 years ago

EDIT :

Can AsyncGPUCallback helps to solve above issue? I think in Unity 2021.3 and above supports asyncgpucallback for android platforms

@karasusan @hiroki-o

karasusan commented 2 years ago

@dhruvaura Please let me explain the situation. First, the Unity Render Streaming package depends on the WebRTC package. To improve streaming performacne, we need to fix issues on the WebRTC package. Last week we released the new version of WebRTC package 2.4.0-exp.8 which contains the performance improvement. And we are preparing to release the new version of Unity Render Streaming.

If you want to try the new version, you can check out the develop branch of this repository. This PR contains changes of updating dependencies. https://github.com/Unity-Technologies/UnityRenderStreaming/pull/714

Apart from that, we know that there are some performance issues for mobile platforms , and we will fix them near future.

dhruvaura commented 2 years ago

@karasusan

Ok thanks for your response. I've also attach profiler two profiler screenshot for you (Before streaming and After streaming), to debug an issue.

I'll try latest release of WebRTC and use 'develop' branch of this repository, will let you know if any significant performance issue fixed or not for Oculus Quest2 device.

dhruvaura commented 2 years ago

Hi @karasusan

I've tried as you suggested to use latest WebRTC and used 'develop' branch of repo to solve performance issue of frame drop of Oculus Quest2 app, but no luck.

After tried latest version of WebRTC and 'develop' branch, still heavily frame drop happening. When streaming not started app runs at stable FPS above 60, but when streaming starts, App FPS goes around 25-35 FPS.

Can you please help how can we fix this issue? By when can be this fix for mobile platforms?

Thanks

karasusan commented 2 years ago

@dhruvaura One question, whet codec are you using? I am not sure the Oculus Quest 2 implements the hardware codec, but probably we should use the hardware codec to improve the performance.

dhruvaura commented 2 years ago

@karasusan Here codec information of Oculus Quest 2 : Codec Info Oculus Quest2

Please check this link page helpful or not.

karasusan commented 2 years ago

@dhruvaura Looks like we can use the hardware codec via Android Media Codec SDK, but I haven't tested the codec on Quest 2. https://forums.oculusvr.com/t5/Quest-Development/Android-MediaCodec/td-p/792547

dhruvaura commented 2 years ago

@karasusan Yes I've checked. Also I debug RTCRtpCodecCapability, m_senderVideoCodecs return 0 codecs, means codec not available in response of TryGetAvailableVideoCodec (StreamBaseSender.cs).

Can you please implement feature for codec, test and see performance of app increase or not. If this issues streaming performance issue get fixed then really helpful to many developer who wants streaming features from mobile, VR devices to browser.

Before URS tried, I also try to implement camera streaming feature app using 'FMETP STREAM V2' that package available at asset store, that package is based on websocket TCP. So faced same issue of performance fps drop when stream start, as well as TCP so packet drop issue also causes additional send-resend packet performance down. Publisher of FMETP suggested me that try to use WebRTC tech of unity might helps.

karasusan commented 2 years ago

@dhruvaura All right. I will check the issue that no codec we can choise on the Oculus Quest 2. Sorry but I can not promise when it will be fixed.

karasusan commented 2 years ago

memo: WRS-378

dhruvaura commented 2 years ago

@karasusan

Ok thanks. Please keep this issue in your pipeline so it'll get fixed earliest.

dhruvaura commented 2 years ago

Hello @karasusan Is this above performance issue get fixed? What is status. Many developers facing this issue, Waiting for solution.

karasusan commented 2 years ago

@dhruvaura No, I haven't fixed.

mgstauffer commented 2 years ago

Hi, fwiw I'm looking for a fix to this issue too.

karasusan commented 2 years ago

Created the issue. https://github.com/Unity-Technologies/com.unity.webrtc/issues/838

Unity-Technologies / UnityRenderStreaming