OnApplicationFocus and local video/audio stream

[michalek-marcin]

Hi,

I want to react when user goes to Hololens main menu during call and pause local video and audio. I use OnApplicationFocus method:

private void OnApplicationFocus(bool hasFocus)
{
    Debug.Log("Application focus state: " + hasFocus);
    _peerConnection.Peer.SetLocalVideoTrackEnabled(hasFocus);
    _peerConnection.Peer.SetLocalAudioTrackEnabled(hasFocus);
}

I created button to switch local video and audio stream and it works when app is active, so pause mechanism works. I can't resume stream when app is active again. I have black screen on second device. Only creating new peer connection works.

Expected result: Video and audio stream are paused in background mode, and resume when app is active again.

Device: Hololens2

WebRtc version: 1.0.2

[djee-ms]

Hi @michalek-marcin,

WebRTC (at least the underlying implementation from Google) was never designed for UWP, so suspending/resuming is not something that would be available out of the box. Therefore we need to decide what to do in this case:

• Temporarily stop sending audio/video to the remote peer (mute), but keep the tracks active.
• Stop and remove the audio/video tracks (which in 1.0.2 requires a renegotiation).
• Other more radical approaches like entirely quit the application

Note: I am assuming that Unity's OnApplicationFocus() is actually equivalent to UWP's app suspend. I think this is the case, but I am not sure, and never tried this to be honest.

It seems you went with the first approach, which is probably the most intuitive. Unfortunately that means that WebRTC continues to do some processing in the background, and in particular receives some audio/video from the network. It simply stops sending some to the remote peer. The problem is that when a UWP app is paused it is expected to stop doing any work, so this is not compatible with the UWP design. Therefore I am not surprised that this does not work. I suspect that UWP is suspending the app completely, and therefore WebRTC cannot process incoming network messages and do other work, so when comes the time to resume the connection has been interrupted for too long and something fails.

https://docs.microsoft.com/en-us/windows/uwp/debug-test-perf/optimize-suspend-resume

When the user switches away from your app, the window is deactivated, but the system waits a short amount of time (about 10 seconds) before suspending the app. This is to give a more responsive experience in case the user switches back to your app rapidly.

In addition of that, when a UWP app is suspended, there is no guarantee that the app will ever be resumed. If you look at the documentation for OnApplicationQuit() there is a comment that says this is not called on UWP, and instead use OnApplicationFocus(false). Basically, there is no way in a UWP app to tell if losing focus means really just losing focus temporarily, or being suspended forever (see this Unity forum thread for example). Therefore keeping a connection alive is probably not possible. This is a limitation of UWP itself.

The best thing I can think of right now, but I didn't try it and I suspect it might fail, is that when the application suspends you remove all the tracks from it, and when it resumes you re-add those tracks. This is probably the best thing we can do, provided that removing all tracks is enough to make WebRTC happy to suspend/resume. To be honest I never really tried, though I experienced the same issue with our TestAppUWP. This is certainly something we should have a look at though, so thanks for opening this issue.

If that doesn't work, then we will need to fallback to the next one, which is completely shut down the peer connection, and recreate it when resuming. Again this sounds overkill but this is actually by design of UWP itself, all apps are doing similar work on suspend/resume. And shutting down the peer connection would actually free some resources, which is always a good thing to do during suspend as per the UWP design guidelines.

[michalek-marcin]

Thank you @djee-ms for a very clear and thorough explanation. I think I must shut down peer connection before app will be suspended and recreate it when app will be focused again. That should work without problems.