[BUG]: H264 encoding problem with NvCodec.

SetoKaiba commented 1 year ago

Package version

3.0.0-pre.4

Environment

* OS:Windows 10
* Unity version:Unity 2020.3

Steps To Reproduce

https://github.com/SetoKaiba/srs-unity Use this project to produce the bug.

Run a srs server with docker, http://ossrs.io/lts/en-us/docs/v5/doc/getting-started
Use Streamer/SceneNormal to stream the unity camera to srs server.
Open http://localhost:8080/console/en_index.html#/streams?port=1985
You will see the stream can't be played
If you open the player before the streamer, the stream can be played. And if you open a player after that, the new player can't be played as well.
If I recompile the libwebrtc with rtc_use_h264=true proprietary_codecs=true ffmpeg_branding="Chrome", and disable the nvcoded encoder. The stream can be played no matter the play is before or after the push.

Current Behavior

No response

Expected Behavior

No response

Anything else?

No response

winlinvip commented 1 year ago

I think the root cause is the NVCodec does not support PLI, which is essential for WebRTC.

SRS is a WebRTC SFU, it works like this:

Publisher --WebRTC---> SRS ---WebRTC--> Player

Note: About the SFU, there is a post about it.

If start publisher before player, the player will request PLI to publisher, because in WebRTC the keyframe might lost or new participants will join the video room which need a new keyframe for decoding.

If publisher does not response for the PLI request, the player never get a keyframe and can't decode the video stream.

SetoKaiba commented 1 year ago

@winlinvip Hi, what's the following lines meaning? Does it matter? The following lines are printed all along with the stream pushing.

[2023-03-20 04:38:04.447][INFO][1][8600d62x] set ts=1345082689, header=32051, lost=32052
[2023-03-20 04:38:04.453][WARN][1][060vu898][11] avc ignore type=1 for no sequence header

SetoKaiba commented 1 year ago

@winlinvip @karasusan I found out the problem. https://github.com/Unity-Technologies/com.unity.webrtc/blob/0bfa9ffc25212ea9a6c6571fdec435973e862dd1/Plugin~/WebRTCPlugin/Codec/NvCodec/NvEncoderImpl.cpp#L445-L450 Change it to the lines below. Add output spspps solve the problem.

        picParams.encodePicFlags = 0;
        if (send_key_frame)
        {
            picParams.encodePicFlags = NV_ENC_PIC_FLAG_FORCEINTRA | NV_ENC_PIC_FLAG_FORCEIDR | NV_ENC_PIC_FLAG_OUTPUT_SPSPPS;
            m_configurations[0].key_frame_request = false;
        }

UPDATE: reminded by @johzzy. The SPSPPS is only needed for keyframe. It solves the problem with verification.

karasusan commented 1 year ago

@SetoKaiba @winlinvip Hi, I am trying NV_ENC_PIC_FLAG_OUTPUT_SPSPPS flag with SRS. I checked it works well. As @SetoKaiba said, the streaming with SRS is failed when not using NV_ENC_PIC_FLAG_OUTPUT_SPSPPS flag.

We will make the PR for solving this issue, do you have any concerns?

winlinvip commented 1 year ago

@winlinvip Hi, what's the following lines meaning? Does it matter? The following lines are printed all along with the stream pushing.
[2023-03-20 04:38:04.447][INFO][1][8600d62x] set ts=1345082689, header=32051, lost=32052
[2023-03-20 04:38:04.453][WARN][1][060vu898][11] avc ignore type=1 for no sequence header

The sequence header is SPS/PPS for H.264, or SPS/PPS/VPS for H.265, it's just a short name.

No sequence header means no SPS/PPS, so it's not able to decode the video stream.

The NV_ENC_PIC_FLAG_OUTPUT_SPSPPS should be set when encoder generate a IDR frames(generally keyframes). And for WebRTC, a keyframe also be sent with SPS/PPS, because the SPS/PPS might change.

I confirmed with @johzzy, it works for me, thanks a lot @SetoKaiba @karasusan

winlinvip commented 1 year ago

We are also working on adding WHIP support to FFmpeg. Please see https://github.com/ossrs/ffmpeg-webrtc/pull/1. After this feature is merged, Unity will be able to communicate with FFmpeg applications.

Unity-Technologies / com.unity.webrtc