Low resolution on some Android phones

holzgeist commented 1 month ago

Hi there,

during development of our app that uses Livekit as the video call framework, we discovered that the video quality is very low on some Android devices. I tracked down at least one bug (and a combination of heuristics combined with some bad luck I guess) in WebRTC, so I'm reporting this here and not in https://github.com/livekit/client-sdk-flutter/, the entry point we use.

The issue

We set the defaultCameraCaptureOptions to VideoParametersPresets.h720_43 (i.e. 960x720) with simulcast enabled. On some Android phones, the sent simulcast layers are 144x192 and 288x384, even on local network with a local livekit server. This is ~16% of the requested resolution, and definitely not suitable for high quality video conferencing.

The deep dive into WebRTC

After ruling out network issues, I started a deep dive into client-sdk-flutter and subsequently into WebRTC itself. Here are my findings (on a Fairphone 5 with Android 13):

Livekit room options:

RoomOptions(
adaptiveStream: true,
dynacast: true,
defaultAudioCaptureOptions:
  AudioCaptureOptions(deviceId: _settings.audioDeviceId),
defaultCameraCaptureOptions: CameraCaptureOptions(
  deviceId: _settings.videoDeviceId,
  params: VideoParametersPresets.h720_43
)));

Camera2Session lists following available capture formats: [4080x3060, 4000x3000, 4000x2250, 3960x1760, 3840x2160, 3264x2448, 3072x3072, 3060x3060, 2700x1200, 2592x1944, 2048x1536, 1920x1080, 1600x1200, 1440x1080, 1280x960, 1280x720, 1200x1200, 1080x1080, 1024x768, 800x600, 800x480, 720x480, 640x480, 640x360, 352x288, 320x240, 176x144]
based on the requested resolution of 960x720, Camera2Session uses the nearest (in terms of total pixels) available format, namely 1024x768: Camera2Session: Using capture format: 1024x768@[7.0:30.0]
The first major reduction of resolution happens on the boundary between Java and C++: https://github.com/webrtc-sdk/webrtc/blob/7ddfc43fcd2e4ba0548258a76fe88d49b34588c5/media/base/video_adapter.cc#L234-L236
- Using the requested pixel count (960x720=691200) as maximum, the video adapter tries to get the input video (1024x768=786432) below that maximum. It only uses 2/3 and 3/4 as factors for efficiency reasons. 3/4 on each axis reduces the number of overall pixels to 9/16 which is ~56%. The current resolution is now 768x576
This downscaling causes simulcast to only send 2 layers instead of 3 because it doesn't reach the 960*540 bucket anymore: https://github.com/webrtc-sdk/webrtc/blob/7ddfc43fcd2e4ba0548258a76fe88d49b34588c5/video/config/simulcast.cc#L82-L106
The simulcast layers are sorted by ascending resolution and originally have scale factors 4, 2 and 1
Finally, due to a bug in the EncoderStreamFactory, the two remaining layers (768x576 and 384*288) are further reduced to 384x288 and 192x144, because they are mapped to the scale factors 4 and 2, not 2 and 1: https://github.com/webrtc-sdk/webrtc/blob/7ddfc43fcd2e4ba0548258a76fe88d49b34588c5/video/config/encoder_stream_factory.cc#L358

The solution(s)

one of
- don't allow Camera2Session to take larger capture format than requested to prevent VideoAdapter to half the video stream later on
- allow the VideoAdapter a slightly increased maximum resolution, similar to max_roundup_rate in simulcast layer selection
maybe add 4:3 tables for simulcast layer selection to compensate for the lower resolution of 4:3 videos compared to 16:9 videos for same height. Currently 4:3 videos tend to fall into lower buckets (with lower layer count)
fix layer mapping bug (PR follows, EDIT here it is: #134)

one more thing

With and without the bug fix, livekit server reports a "Good" connection quality for that phone despite being on the same network. On a different phone, that has 960x720 in its native capture formats, layer selection works and connection quality is "Excellent". This is likely a bug in livekits scorer that I will investigate separately. I still wanted to mention it here because it was the reason for my investigations

davidliu commented 1 month ago

Hey, thanks for the report and PR! I'll take a look at this over soon.

davidliu commented 1 month ago

With and without the bug fix, livekit server reports a "Good" connection quality for that phone despite being on the same network. On a different phone, that has 960x720 in its native capture formats, layer selection works and connection quality is "Excellent". This is likely a bug in livekits scorer that I will investigate separately. I still wanted to mention it here because it was the reason for my investigations

We haven't yet looked into this deeply, but someone mentioned that because the server expected three layers and got only two, the score could be dinged.

holzgeist commented 1 month ago

We haven't yet looked into this deeply, but someone mentioned that because the server expected three layers and got only two, the score could be dinged.

That was me after some more digging :smiley:

davidliu commented 1 month ago

Looks like this might be a flutter-webrtc issue, as they use the adaptOutputFormat functions with the requested resolution. This causes the initial downscaling pointed out at the Java/C++ layer when the actual capture format is larger.

@cloudwebrtc I think it might be good to just remove the adaptOutputFormat calls here, or at least adding an option to skip the adaptOutputFormat call so that frames are sent to the native layer unadapted (the capturer should be capturing close to the target anyways). Alternatively, we could bump in the values passed into adaptOutputFormat by some factor (say 1.5) to allow for the rounding up of the camera capture format. Android SDK doesn't use this method, so should be fine to remove though.
4:3 buckets probably not needed, since the Livekit sdks determine by the longest side whether we will use 3 layers or not (which means 4:3 will have more pixels than the 16:9 with the same longest side), and we require at least 960 on the longest side, which lines up with the WebRTC buckets.
I think we actually can't use the PR, since this affects the layer ordering information and the SFUs are sensitive to that. However, I think fixing up the flutter-webrtc code to accurately get the actual video camera dimensions used will address the issue of simulcast layer limiting and thus make this not needed.

There's another alternative we can use (which also addresses point 2), which is to simply turn off the native simulcast layer limiting so that we don't run into this mismatch in the first place.

@cloudwebrtc @hiroshihorie we can turn off the simulcast layer limiting by adding WebRTC-LegacySimulcastLayerLimit/Disabled/ as a field trial. Note, for smaller resolutions (under 480px longest side), the native code limited to 1 layer only, and we should similarly follow suit and set a minimum longest side to send 2 layers regardless of whether we go with disabling the field trial or not.

holzgeist commented 1 month ago

Hi @davidliu thanks for your thorough review and quick fixes in the linked PRs. I tested them and they fix all the described problems in my bug report above :heart:

Even though it's not needed anymore, can you elaborate on the layer ordering? I don't see where I change it. The resulting layer array is the same as before, but has correct resolutions/scaling factors set.

davidliu commented 1 month ago

@holzgeist looking at it again, I think I just brain-farted and thought it reversed the order. The ordering should be fine. I think we can actually use it actually. I'll need to do a little more digging and testing though

holzgeist commented 1 month ago

@davidliu thanks :) I'll close the issue for now though, because the problems are fixed and the PRs merged. Let's continue discussion on PR #134 (if needed)

webrtc-sdk / webrtc