[BUG]: Unity does not negotiate hardware supported codec

[jwielebnowski]

Package version

3.0.0-pre.6

Environment

* OS:Windows
* Unity version:2020-2023

Steps To Reproduce

Setup and run your webrtc enabled game with your package and with your client webserver.exe receiver.

Current Behavior

The game consistently utilizes the VP8 codec, which lacks hardware acceleration. As a result, the streaming quality suffers, displaying significant pixelization

This is from the webrtc-internals:

inbound-rtp (kind=video, mid=1, ssrc=4196375540, rtxSsrc=3366492407, decoderImplementation=libvpx, powerEfficientDecoder=false, [codec]=VP8 (127), id=IT01V4196375540)

Expected Behavior

The webrtc connection should negotiate with the hardware supported codec, preferably H264. I have tried the following to force the game to use H264 codec and it did not work:

Order the payloads on the client offer and the unity answer so the H264 codec should take precedence

Remove all other codes but H264 from the client offer

The only way I was able to 'fix' the issue was to override VP8 and VP9 payloads with H264 and the game connected with the client using H264 codec.

Anything else?

[FabienDanieau]

I've observed a similar issue with a client that create a SDP offer with VP8 and H264 (https://github.com/aiortc/aiortc/blob/main/src/aiortc/codecs/__init__.py). It is hardcoded, although I send a H264 video. So the Unity client agrees on VP8 codec during the negociation and I can't see anything. My fix is to remove the VP8 codec from the offer which should not be there in the first place.

But I tried to fix this on the Unity side first. If I set the preferred codec as H264, VP8 is still selected. The preference seems ignored.

[jwielebnowski]

@FabienDanieau I cannot remove the VP8 from the offer since I get an error about the send/receiver setting not being correct. I could not find work around it. The only thing I could do was to replace V8/V9 with H264, and even if it connected with a different codec all was ok. But it is not a clean fix. It looks like a defect to me.

[FabienDanieau]

I agree. My point is that the preferred codec is not negotiated which looks like a bug.

[karasusan]

@jwielebnowski Hi, the "ChangeCodec" sample in contained in the WebRTC package sample. https://github.com/Unity-Technologies/com.unity.webrtc/blob/main/Samples~/ChangeCodecs/ChangeCodecsSample.cs#L175-L190

This sample shows the way of filtering codecs on sender side. This process should be ran before signaling.

[jwielebnowski]

@karasusan, I'm experiencing some issues when trying to set the codec preferences. Here's the code I'm using:

       var capabilities = RTCRtpSender.GetCapabilities(TrackKind.Video);
            var availableCodecs = capabilities.codecs.Select(item => item).ToArray();
            var availableCodecs = capabilities.codecs.Where(codec = !excludeCodecMimeType.Contains(codec.mimeType)).ToArray();

            availableCodecs = availableCodecs
            .OrderByDescending(obj => obj.mimeType == "video/H264")
            .ThenBy(obj => obj.mimeType)
            .ToArray();
            RTCRtpCodecCapability preferredCodec = availableCodecs[0];
            var codecs = new[] { preferredCodec };

            RTCRtpTransceiver transceiver = pc.GetTransceivers().First();
            RTCErrorType error = transceiver.SetCodecPreferences(codecs);
            // RTCErrorType error = transceiver.SetCodecPreferences(capabilities.codecs);
            if (error != RTCErrorType.None)
            {
                Debug.LogErrorFormat("RTCRtpTransceiver.SetCodecPreferences failed. {0}", error);
            }

When I try to run this code before the following block:

            var setRemoteDescriptionOp = pc.SetRemoteDescription(ref offer);
            WaitForAsyncOperation(setRemoteDescriptionOp);

I find that pc.GetTransceivers() returns an empty list. However, if I execute it after SetRemoteDescription, I encounter an InvalidModification error at this line: RTCErrorType error = transceiver.SetCodecPreferences(codecs);.

I've tried various approaches, including getting and setting codecs without making any modifications, but to no avail. Could this possibly be related to how I'm obtaining capabilities with RTCRtpSender.GetCapabilities(TrackKind.Video)?

Do you see any issues with my implementation? Any guidance would be greatly appreciated.

[karasusan]

@jwielebnowski It looks almost identical to the "ChangeCodec" sample.

[jwielebnowski]

Unity Answer.txt Browser Offer 1.txt Browser Offer 2.txt

@karasusan In the example, the offers were created on the browser client, and the codecs were subsequently sorted so that the H264 codecs were placed at the top. The Unity Answers were generated based on the example you provided. For Browser Offer 1, the selected codec was VP8. For Browser Offer 2, the selected codec was AV1. Despite these configurations, I still can't get H264 to be selected. If on the client I just send the unmodified offer, always VP8 codec is selected. It looks like I do not have a control over it what is happening on the Unity side. Can you confirm that the Unity plugin is doing the right thing?

[jwielebnowski]

@karasusan Firefox will always connect to the Unity stream using the VP8 codec. Can you confirm that there is no workaround for this?

[karasusan]

@jwielebnowski I checked H.264 codec with Firefox. I'd like to know how to replicate your issue.

https://github.com/Unity-Technologies/com.unity.webrtc/assets/1132081/388e9eec-3a9e-4f45-9127-b0b75de91321

[jwielebnowski]

@karasusan here are screen captures. The game is using unity plugin and unity sample web client. When using Firefox it always connects with VP8.