[BUG] Visual artifacts when using hardware encoding

[LukasKastern]

Describe the bug I've noticed visual artifacts when using hardware encoding. I wasn't able to reproduce these artifacts using software encoding.

These artifacts are easier to reproduce with complex scene setups, different colors, and frequent color transitions. I was also able to reproduce these artifacts with less complex setups by simulating packet drop in combination with latency. The higher the latency the less packet drop was needed.

To give an example these screenshots were taken with 100ms inbound lag and 5% inbound packet drop. Inbound meaning the packets that were sent to the browser which displayed the video stream.

To Reproduce Steps to reproduce the behavior:

1. Get one of the RenderStreaming sample projects
2. Enable hardware encoding in the RenderStreaming MonoBehaviour
3. Click play and connect through the browser
4. Simulate an environment with an inbound latency of 100ms and 5% packet drop (I've used clumsy for that)
5. You should be able to see similar artifacts to those in my screen shots

Environment (please complete the following information):

• OS: Windows
• Browser chrome
• Unity Version 2019.4.9
• Package version 2.1.2-preview

[WaltzBinaire]

hey guys!

We can confirm those finding by reproducing the flow of @LukasKastern - its best visible when moving sideway or rotating the camera a lot. Only visible in hardware encoder.

[autr]

Effects seem to be “data-moshing” where intra-frames are being dropped from the packet loss.

Would be great to have more of the libwebrtc and NVENC settings exposed so that we could test different keyframe intervals, b-frames etc - it’d be much preferable to have dropped frames / lag over glitch artefacts.

The software encoder is also using VP8 whereas hardware is H264, so that could be related. There’s a comment here about implementing other codecs (including VP8):

https://github.com/Unity-Technologies/com.unity.webrtc/blob/989e7b2ad495a6c35b99b482b7b1317e84c77110/Plugin~/WebRTCPlugin/UnityVideoEncoderFactory.cpp#L43

[WaltzBinaire]

hey @karasusan ,

would it be possible to expose more encoding settings - basically as many basic settings as possible. For example the ones @autr mentioned ? Then we would be able to help and ping back about our research how to lower the amount of artefacts.

[autr]

Hi @karasusan - here's some more details on the bug. I made an example project here from the default UnityRenderStreaming HDRP example (adding particles and colour cycling to trigger intra-frames).

https://github.com/autr/UnityRenderStreaming-glitches

On a local network with a MacBook as client I’m using Apple’s inbuilt Network Link Conditioner. If I set it to 2% packet loss or upwards then the keyframe / data-moshing glitches occur (without changing delay or bandwidth settings).

Another strange effect is that also after doing this the stream never recovers and continues to do these keyframe glitches until the browser is refreshed.

[autr]

Here is some additional research to compare with Unreal's PixelStreaming (which doesn't have the bug). The screengrab is of the NVIDIA API settings that are used in each, but otherwise the implementations are quite similar:

Both encoders are forcing an IDR frame when receiving webrtc::kVideoFrameKey, however Unity has a bitwise operator which could be accidentally causing an Intra, rather than IDR, frame: picParams.encodePicFlags = NV_ENC_PIC_FLAG_FORCEIDR | NV_ENC_PIC_FLAG_FORCEINTRA;

Unity seems to be calculating an average bitrate from the initial incoming width and height during initialisation, then later allows this to be changed via the UpdateSettings method, so possibly this could be related to the unusual ramping behaviour observed in chrome://webrtc-internals:

# init
averageBitRate = (static_cast<unsigned int>(5.0f *, nvEncInitializeParams.encodeWidth *, nvEncInitializeParams.encodeHeight) / (m_width * m_height)) * 100000
# update
if (nvEncConfig.rcParams.averageBitRate != m_targetBitrate)
{
    nvEncConfig.rcParams.averageBitRate = m_targetBitrate;
    settingChanged = true;
}

Unity currently uses a deprecated rate control mode NV_ENC_PARAMS_RC_CBR_LOWDELAY_HQ. Unreal exposes 3 rate control modes, and Bitrate and MaxBitrate parameters which are set to averageBitRate and maxBitRate. The modes are:

• NV_ENC_PARAMS_RC_CONSTQP requires NV_ENC_RC_PARAMS::constQP to be set
• NV_ENC_PARAMS_RC_CBR uses only NV_ENC_RC_PARAMS::averageBitRate
• NV_ENC_PARAMS_RC_VBR uses NV_ENC_RC_PARAMS::averageBitRate and NV_ENC_RC_PARAMS::maxBitRate

And during initialisation and update, additional flags for minimum quantisation (compression) are set:

    # custom params

    rateControlMode = <config.RcMode> = NV_ENC_PARAMS_RC_CONSTQP || NV_ENC_PARAMS_RC_VBR || NV_ENC_PARAMS_RC_CBR
    maxBitRate = <config.maxBitRate>

    # additional flags

    enableMinQP = true
    minQP = { 20, 20, 20 }

I'd like to try adjusting some of these settings but am currently unable to run a re-compiled plugin: https://github.com/Unity-Technologies/com.unity.webrtc/issues/216

Help much appreciated!

[karasusan]

@autr I have never known that the NV_ENC_PARAMS_RC_CBR_LOWDELAY_HQ parameter is deprecated. I'll investigate it.

[karasusan]

I am trying to reproduce the issue but I can't yet. I am using the latest version of the package (this is a develop branch, not released yet). I will try to reproduce it again with 2.1.2-preview version.

[karasusan]

I reproduced the glitches in my environment with "packet loss 10%" When using VP8 encoder, video quality looks good but the framerate is low. It might be difficult to tune the options of the encoder for everyone to convince.

HW encoder on (H264)

HW encoder off (VP8)

[karasusan]

@LukasKastern @autr, @WaltzBinaire I posted above. Do you have any comments?

[autr]

Hi @karasusan - adding particles or animating colours to the scene will more reliably trigger intra-frames and the glitch. Please check out the example here which is a modification of the HDRP example:

https://github.com/autr/UnityRenderStreaming-glitches

There is also an experimental fork I made of com.unity.webrtc which makes available various recommended NVIDIA settings. It's via the 2.1.3 branch because I couldn't yet get the downloaded libwebrtc or re-compiled libwebrtc to work with develop:

https://github.com/autr/com.unity.webrtc

List of parameters exposed are:

And so far I had some success by tweaking Intra Refresh Period and Intra Refresh Count according to NVIDIA docs. This is an “error resilience mechanism” however, so I don’t think it’s fixing the root problem so much as just helping to minimise it.

Frame-rate seems like it might be the issue - which is also something Unreal has custom logic for. In the fork there is a script RenderStreamDebugger.cs which will display the incoming FPS and bitrate from the webrtc callback; which is called after a client connection is made, and is independent of the encoder’s NV_ENC_INITIALIZE_PARAMS::frameRateNum encoder initialisation.

This floats around ~60fps regardless of how bad the connection is, which is consistent with the webrtc::codec_config::h264::maxFramerate . However when modifying webrtc::maxFramerate or NV_ENC_INITIALIZE_PARAMS::frameRateNum, this seems to cause more glitches (though I haven’t had much time to investigate).

Hazarding a guess - these framerates need to be synchronised to something lower, and exposed as a controllable target or max framerate - because the glitch may simply be that the implementation is trying to push out too many frames. In particular, Unreal has custom logic which will drop FPS to preserve bitrate based on its own calculations during the callback.

Some additional notes (inc. on the Unreal implementation) can be found here:

https://github.com/autr/unity-notes

[karasusan]

@autr I have to thank your great work and I would like to respond it. I agree your conclusion that the framerate is the cause of the issue. We need to add the operation to adjust number of frame to encode.

In our approurch, this is different from Unreal, the way deligates encoding process to worker threads provided by libwebrtc. Already we started to develop to fix this issue, but affected area is huge so it has not been completed yet. You can see our change for this issue on this branch. (DX12 is not corresponded yet.) https://github.com/Unity-Technologies/com.unity.webrtc/tree/experimental/encoder-thread

I am afraid that the release date of this fix is not determined.

[kayy]

I am not yet sure, but it it appeared to me as the supplying x-google-min-bitrate, x-google-start-bitrate and x-google-max-bitrate decreases the effect slightly.
Anyway I am looking forward to a fix

[karasusan]

We are preparing to merge this change for fixing this issue. https://github.com/Unity-Technologies/com.unity.webrtc/issues/205#issuecomment-715140844

But I know this change will result in decreasing concurrent users when using the Geforce series GPU. Because the number of encoders increases by concurrent users. As for now, it is decided by the number of cameras.

For resolving above the issue, need to use Simulcast with the third-party SFU, but we have insufficient knowledge about it yet. I am sorry for the delay but we need more time to familiar with this technology and publish documents to take measures.

[scode-cmd]

We are preparing to merge this change for fixing this issue. #205 (comment)

But I know this change will result in decreasing concurrent users when using the Geforce series GPU. Because the number of encoders increases by concurrent users. As for now, it is decided by the number of cameras.

For resolving above the issue, need to use Simulcast with the third-party SFU, but we have insufficient knowledge about it yet. I am sorry for the delay but we need more time to familiar with this technology and publish documents to take measures.

Any update on this issue? It's been more than one year. We have a private demo in front of important industry stakeholders in two weeks and we are wondering if we can get a branch with the fix.

Disabling hardware encoding does not solve the issue (on an RTX 3090, software encoding produces completly corrupt results)

[karasusan]

@scode-cmd You mean there is glitch artifacts when using software encoder? The issue might be the same problem of this ticket. https://github.com/Unity-Technologies/UnityRenderStreaming/issues/530

[karasusan]

memo: WRS-171

[yvanzine]

I believe this may be a long standing bug in hardware H264 encoder implementation in com.unity.webrtc.

I tried modifying H264 encoder by controlling for P-frames etc, to no avail.

I tried to mitigate the "smearing" effect by lowering rendering framerate to help allocate bandwidth better, but at 1-2Mbps bandwidth there is still smearing which makes me feel like there is a bug in the current H264 encoder implementation.

I am attaching the videos taken at low framerate (5fps) and bandwidth (1-2Mbps) to compare incorrect (smearing) and correct H264 (drop in resolution while in motion but no smearing) encoder implementations:

SMEARING with com.unity.webrtc: https://user-images.githubusercontent.com/19309404/156064568-012af0f6-8735-4837-a105-d9bc3fc0a330.mp4

NOT SMEARING with an alternative H264 encoder: https://user-images.githubusercontent.com/19309404/156064703-e3bf0cc5-dcda-4ede-91d7-26e96eaed2fd.mp4

[karasusan]

@yvanzine Hi, thanks for sharing the detail. We are going to investigate the issue this month.

[yvanzine]

@karasusan I wanted to point out, that if you take a screenshot of the "smearing" frame, It looks like invalid buffer, not as intermediate frame of h264 encoded frame. There is possibly a bug of grabbing and drawing that frame at the wrong time.

[gtk2k]

Will this issue be fixed (to some extent) in an update scheduled for April?

[karasusan]

@gtk2k Yes.

[gtk2k]

@karasusan thx! As a test, when I build and run feat / nvcodec, I get the error in the example with local offer. (Failed to set local video description) (Windows 10/11)

(日本語で) feat/nvcodec を試しにビルドして実行してみましたが、local offer設定時にエラーになりました。 (Failed to set local video description) (Windows 10/11)

[karasusan]

@gtk2k もう少し詳しく状況を教えていただけると何かお答えできるかもしれません。

[gtk2k]

@karasusan すいません。Runtime/Scriptsのほうを更新し忘れていました。 Runtime/Scriptsを更新すると出なくなりました。

ちなみになのですが、Scriptを更新すると WebRTC.Initialize()の引数にハードウェアかソフトウェアかの設定がなくなっていました。 Transceiverにハードウェアエンコードをサポートしたコーデックを設定することでハードウェアエンコードになるという認識で間違いないでしょうか？

[karasusan]

@gtk2k はい、コーデックを変更する箇所を Initialize から Transceiver の API に変更しています。

[gtk2k]

@karasusan 了解です

[karasusan]

We changed the implementation of the native code related this issue on the develop branch. I already updated the native plugin and you can check whether reproducing the issue or not if you check out the branch. Please let me know if you reproduce the issue with the develop branch. Thank you.