Blocky video from hardware H.264 encoder on HL2

[fibann]

On HL2 some configurations produce a stream that is much "blockier" than the result of the same configuration on HL1 or PC. A possible reason is that the HL2 encoder uses a higher QP range than other encoders by default. WebRTC has a way to specify the QP explicitly but the encoder implementation ignores the value at the moment.

[fibann]

Fix https://github.com/webrtc-uwp/webrtc-windows/pull/74 has been integrated by 6d9b8a79abaed3d13dcd7443277bc3859014290c. Now it's possible to mitigate this issue by setting a QP lower than the default (51).

After 4e18316d03f1303ebc5402611a773ba8f4d62ff6 this can be done by adding x-google-max-quantization=<value> to PeerConnection.PreferredVideoCodecExtraParamsLocal on the sending endpoint, or to PeerConnection.PreferredVideoCodecExtraParamsRemote on the receiving endpoint, before starting the streaming.

~This can be done by:~

• ~creating the SDP offer from the endpoint that is going to receive the video, and~
• ~adding x-google-max-quantization=<value> to PeerConnection.PreferredVideoCodecExtraParams before creating the offer.~

~The setting seems to only be applied to the encoder when appearing in the remote description, and at the moment PreferredVideoCodecExtraParams is only added to offers (not answers) - thus it only has effect if used in an offer that comes from the receiving endpoint. I suppose we could extend the extra-params mechanism to apply to answers too, and remove this limitation.~ (needed before 4e18316d03f1303ebc5402611a773ba8f4d62ff6)

Note that setting a low max QP makes the framerate drop in scenes with movement going on. In the scenario I tested (1280x720@30, 1Mbps) a value of 45 seemed to bring some quality improvement while maintaining a stable framerate, but YMMV.

[kspark-scott]

Thanks Filippo (@fibann). We can try this out, though it may take a few days before I have the time to fit it in. It will require a change to how we do signalling as our calling is not symmetric: calls can be initiated from HoloLens only. So currently the HoloLens creates the SDP offer and then sends but does not receive video. If I understand you correctly, we'll have to reverse that and have the call receiver create the offer and the HoloLens create the answer. We do already have call/answer signal messages that are separate from the SDP offer/answer so it's not too disruptive, but is more than just adding an extra option to the codec params.

[eirikhollis]

@fibann How much of a difference should it make? I'm setting the quantization value before the offer with

PreferredVideoCodecExtraParams = "x-google-max-quantization=45"

but even with values ranging from 10, 20, 30, ..., 80, I see no difference in blockiness. Is there a way I can verify that it has been set? CPU usage on HL2 stays consistent throughout the range as well.

[djee-ms]

@eirikhollis did you recompile Google's implementation on master? Because we didn't roll out 1.0.3 NuGet packages with that fix yet (it's underway), so if you're using NuGet and not a local build of the core dependencies (external/webrtc-uwp-sdk) then you don't have that fix. Otherwise please wait a few more days for the new NuGet packages.

[eirikhollis]

On master branch and am building the MixedReality-WebRTC project from source. Is it in there?

[djee-ms]

No. The fix comes from a change upstream in webrtc-uwp-sdk, not in this project. You have to either wait for new NuGet core dependencies (not the MixedReality-WebRTC NuGet packages) or rebuild them from scratch, which is fairly involved : https://microsoft.github.io/MixedReality-WebRTC/manual/building-core.html But I would recommend against it if you never did it. That might take a few days on first try to get all dependencies setup correctly on a new machine and understand the process.

[eirikhollis]

Thanks for the explanation, will wait a few days then :+1:

[fibann]

@kspark-scott, commit 4e18316d03f1303ebc5402611a773ba8f4d62ff6 extends the filtering to incoming SDP requests, which means that you can now set the parameter on HoloLens and it will work even if it is the one creating the offer.

[kspark-scott]

@fibann, fantastic, thanks. I hope to get a chance to try this out on Monday (I'm set up to build the core dependencies locally).

[kspark-scott]

@fibann, I was able to try the new max-quantization parameter today and can confirm that it appears to be working as expected in the context of our app without requiring any changes to our signalling. Thanks for your efforts!

The outcome is a big improvement, though not quite ideal because there is no overlap where blockiness is completely removed without compromising frame rate.

Here is an informal assessment based on testing at 1280x720@30. I tried both 1mbps and 2.5mbps and saw no clear difference between the two after 30 seconds or so, though I'm not 100% certain I'm setting the bitrate correctly.

QP=45: I did not see a noticeable improvement in blockiness, ~30 fps QP=40: noticeable improvement but blockiness still bad enough to be distracting, 25-30 fps QP=38: maybe the sweet spot for us, with significantly reduced blockiness and >20 fps QP=35: blockiness noticeable only when you look for it, but fps reduced to 10-15 QP=30: blockiness virtually eliminated, but fps=5

So this is definitely an improvement, though not yet on par with HoloLens 1 @ 896x504. Is it still reasonable to expect (eventually) the same quality on HL2 @ 1280x720, even with more than double the pixels/frame at that resolution?

[Dryra]

Hello, is the 1.0.3 Nuget Package with the fix already rolled out?

[djee-ms]

No, we didn't roll it yet, we paused it as we were waiting both for some additional last-minute code change and for Scott's feedback, and then this fell through the cracks. I will have a look next week to backport that change to the release/1.0 branch and roll some packages. @Dryra do you expect to need this? I think @kspark-scott eventually used a lower resolution to work around the issue instead, which both yields better quality and reduces the CPU load on HL2, so proved a better approach than tweaking QP if you can.

[Dryra]

@djee-ms thank you for the prompt reply! Yes I need this until next week, I will try to work around it for the moment with setting the resolution down until you roll to the release/1.0.

[kspark-scott]

I can confirm that we did indeed go with the lower resolution (896x504). We often use relatively low bit rates (generally 1 mbps but often 512 kbps and even 128 kbps). At low bit rates, the quantization produced by encoding double the number of pixels in the same number of bits is significantly worse than the quality penalty of the lower capture resolution. So even when the quantization parameter can be controlled directly we are likely to stick with the lower resolution because it will still look better at the lower bit rates.

[Dryra]

Thank you @kspark-scott. So if I understood correctly, in the unity integration project, I change the width to 896 in the LocalVideoSource.cs script // Holographic AR (transparent) non-x86 platform - Assume HoloLens 2 videoProfileKind = WebRTC.PeerConnection.VideoProfileKind.VideoConferencing; width = 896; // Target 1280 x 720 just like the one set for the HL1. I did this and tested it with the HL2, but the android client (remote peer in my case) complained that it did not get any video Tracks from the remote stream, but when I change the width back to 1280 it works correctly, do I have to change anything in the SDP message I'm sending or receiving?

[fibann]

@Dryra on HL2 the 896x504 profile seems to be under the BalancedVideoAndPhoto kind, not VideoConferencing. So if you change the profile kind too it should work.

[Dryra]

Hello @fibann, your suggestion works, it's still a little bit blocky but way better than before. We will have to live with this workaround for now until a new fix is out. Thank you!

[djee-ms]

@Dryra, @eirikhollis : the v1.0.3 packages are out on nuget.org with that QP fix. I will merge the change to master now.

[fibann]

528 adds the possibility of choosing an encoding profile higher than the default (Baseline). This brings better quality for the same bitrate, at the expense of compatibility (some devices/encoder implementations might support Baseline/Constrained Baseline only). On devices with a hardware encoder - which includes HL2 - the improvement should be essentially free in terms of performance.

From the few tests I have run, on HL2 High profile brings a net quality gain compared to the default, while still falling short of the HL1 quality, so it should be considered a partial mitigation.

Note that the same change adds a way to specify the Max QP directly without having to resort to modifying SDPs.

[fibann]

An update on this. We have done extensive investigation and the difference in quality seems to be related to details in the encoder implementation, rather than to configuration issues. Windows version 19041.1384 (available on Dev/Beta Windows Insiders rings) contains improvements to the encoder implementation that increase quality at low bitrates in some configurations.

In order to enable the improved path, the H.264 encoder must be configured to

• Profile: High (ConstrainedHigh and Main should work too)
• Rate control mode: VBR
• Common Quality: 33 (or similar; this is mostly to avoid VBR overshooting the target bitrate, YMMV).

In the MR-WebRTC C# API , on the latest master, this can be done with:

PeerConnection.SetH264Config(
  new PeerConnection H264Config
  {
    Profile = PeerConnection.H264Profile.High,
    RcMode = PeerConnection.H264RcMode.VBR,
    Quality = 33
  });

This configuration should generate similar encoding quality to the default configuration (Baseline, CBR) on HL1.

[eirikhollis]

@fibann I'm currently testing latest master with the settings you've provided above and it looks promising. I'll give some more feedback when I've tested with several HL2's connected in the same call with varying level of quality settings so that I can spot the differences easier.

I have three questions though

• The max quantization level isn't set in the code. Is that less important now? I have it set to 45 in my tests.
• Do you call PeerConnection.SetH264Config once for every PeerConnection, or just one time throughout the application lifetime, no matter how many PeerConnections are created? Is there a best time to call the method if it should only be called once?
• Enabling and disabling video seems to affect the HL2 video stream in a very negative way, reducing the framerate to 0.1-0.5 fps when enabling the video again. Have you tested this? I'm enabling/disabling the video by toggling the LocalVideoTrack.Enabled property. Destroying and creating the PeerConnection object seems to fix it, so the problem is only present until the HL2 republishes its local video feed. Having the video disabled for a longer period of time seems to have a more severe effect.

[fibann]

@eirikhollis

The max quantization level isn't set in the code. Is that less important now?

That works as before - it sets a lower bound on individual frame quality, possibly at the expense of framerate. It acts independently from the other options.

Is there a best time to call the method if it should only be called once?

The configuration is global so you can call it only once - no best moment, you can do that at application start.

Enabling and disabling video seems to affect the HL2 video stream in a very negative way

Can you open a new issue with version/repro steps? Does this happen only when using PeerConnection.SetH264Config?

[eirikhollis]

Can you open a new issue with version/repro steps? Does this happen only when using PeerConnection.SetH264Config?

Tested quickly with just commenting out the one line of PeerConnection.SetH264Config I had, and yes, that fixed the issue. Tried building two separate builds where I adjusted where the PeerConnection.SetH264Config command went, but both resulted in low fps on LocalVideoTrack.Enabled = true after first disabling it.

I'm using my own custom solution for this, and the only thing that is equal between my and the sample project you provide are the mrwebrtc.dll and Microsoft.MixedReality.WebRTC.dll. I'm building from Unity 2019.4.11 with min SDK 17763 and target SDK 19041.

Do you still want me to open an issue on this?

Edit: What I can say is that the images with reduced fps are veeery crisp, and resembles having set the Max Quantization value to something low, such as 20.

[fibann]

Do you still want me to open an issue on this?

No thanks, let's track it here since it seems related to SetH264Config.

[eirikhollis]

Any updates on this?

[fibann]

I haven't got around to investigate this but I have a few ideas - the encoder tends to exhibit the same behavior in VBR when frames are dropped, so it might be reacting the same way to the period while the track is disabled (as if a lot of frames were dropped in sequence). Recreating the PeerConnection will reset the encoder which explains why it works after that. It might be possible to trigger the reset by simply changing the Transceiver direction (which is less disruptive), I am not sure but it's worth trying if that makes things easier.

[astaikos316]

@kspark-scott i have been trying to run some tests with low bandwidth just between 2 PC’s right now and cannot get reliable performance when setting bitrate to 128kbps. I started logging frames encoded and dropped and see that every few seconds all frames are dropped on the receiver side. This happens at a pretty consistent interval (every few seconds) and I am not sure where to look. Any help would be greatly appreciated. Thank you in advance.

[kspark-scott]

@astaikos316 What capture resolution and frame rate are you using? If you make impossible demands of the encoder -- e.g. "please encode this 1920x1080@30 fps capture into a 128kbps stream" -- that will definitely produce reliability issues. That said, low bandwidth modes have always just worked for us so I haven't had the need to dig more deeply and develop more understanding.

Though we don't support it in our product, I have done calls with audio+video outbound from HoloLens and audio only inbound with a 64kbps bandwidth cap with no reliability issues. The caveat is that I had to use a very small capture resolution and frame rate to pull it off.

For the record, we set bandwidth in two places. When initializing peer connection:

uint codecBitRate = (uint)(CallConfiguration.VideoBitRate + CallConfiguration.AudioBitRate) * 1024;
m_peerConnection.SetBitrate(minBitrateBps: null, startBitrateBps: codecBitRate, maxBitrateBps: codecBitRate);

and we modify the SDP we get from WebRTC before delivering the offer and answer by finding the m=audio or m=video sections and adjusting the b=AS: value. That second part is legacy code we used before MixedReality-WebRTC existed and may no longer be needed, but we still do it and the behavior matches our expectations.

[astaikos316]

@kspark-scott i have been trying with at resolution of 320x240 at 15 fps but continuously getting all frames dropped. When using setbitrate, when I look at the sdp there are no b=AS lines at all. I have tried running setbitrate when initializing peer and at other points (before call starts, etc). Using Mixed Reality WebRTC I cannot find where I can grab the SDP to modify it.

[kspark-scott]

@astaikos316 we update the SDP in the PeerConnection.LocalSdpReadytoSend event handler before sending it to the peer. There generally won't be a b=AS line. If there is, you replace it, if there is not, you add it. The general procedure (the one we use at least) is to break the SDP into lines by splitting on '\n', then iterate through the lines:

• locate the m=audio and/or m=video line
• skip to the next line
• skip any subsequent i= or c= lines
• if if you are on a b= line, replace it, otherwise add it
• rejoin the lines into a single string and send to peer

[spacecheeserocks]

i have been trying to run some tests with low bandwidth just between 2 PC’s right now and cannot get reliable performance when setting bitrate to 128kbps.

128kbps sounds far too small if you're trying to get any video, even at a low resolution!
128kbps is just about enough for low-medium quality audio (depending on encoder).

obviously keep testing, but I would probably recommend at least 1mbps for a low quality video stream.

[astaikos316]

@kspark-scott I managed to get bi-directional audio/video working as expected now at 128kpbs after using both setbitrate as well as modifying the SDP messages appropriately. One odd thing I am still noticing though is that even though I am sending frames at 15fps, when I monitor the received frames via GetStats API, the receiver is saying they are receiving 30 fps.