"Could not parse H265 SPS Unit"

[bchah]

Describe the bug OME has supported HEVC input via SRT for some time now, however in my testing across multiple software and hardware streams the provider is unable to accept the stream and always throws this error in the logs:

[2023-04-01 15:01:01.898] E [SPSRT-s9999:342] MpegtsProvider | mpegts_stream.cpp:168  | Could not parse sps
[2023-04-01 15:01:01.900] E [InboundWorker:349] MediaRouter | mediarouter_stream.cpp:598  | Could not parse H265 SPS Unit

An additional clue is that the stream statistics always seem to report the resolution of HEVC streams as "0x0". This behaviour is consistent across many different stream encoders. I have tried so far with:

• OBS on macOS (Apple VT HEVC Encoder)
• OBS on Windows (Nvidia NVENC HEVC)
• Larix Broadcaster on iOS
• Magewell Ultra Encode / Ultra Stream
• Various models of URAY Encoder
• Various models of TERADEK Encoder

To Reproduce Send an HEVC stream into an SRT or WHIP provider on any version of OME. The error in the logs is always the same.

Expected behavior OME should accept the stream and apply the OutputProfiles specified to create H264 for the viewers.

Logs ovenmediaengine.log

Server (please complete the following information):

• OS: Docker on macOS (or Linux, or Windows)
• OvenMediaEngine Version: 0.15.7 (or any other version)
• Branch: master (or dev) (This problem has been consistent for a long time)

Additional context Here are the OutputProfiles from my server.xml - the expectation is that given an HEVC/AAC input these would transcode H264/OPUS for the WebRTC Publisher and JPEGs for the Thumbnail Publisher. But so far the SRT Provider has been unable to process the inputs, so these probably aren't even getting hit yet:

 <OutputProfiles>
                                                <OutputProfile>
                                                        <Name>stream</Name>
                                                        <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                                        <Encodes>
                                                                <Video>
                                                                   <Codec>h264</Codec>
                                                                        <Bitrate>4000000</Bitrate>
                                                                        <Profile>baseline</Profile>
                                                                        <Preset>faster</Preset>
                                                                        <ThreadCount>4</ThreadCount>
                                                                        <BypassIfMatch>
                                                                                <Codec>eq</Codec>
                                                                        </BypassIfMatch>
                                                                </Video>
                                                                <Audio>
                                                                        <Codec>aac</Codec>
                                                                        <Bitrate>128000</Bitrate>
                                                                        <Samplerate>48000</Samplerate>
                                                                        <BypassIfMatch>
                                                                                <Codec>eq</Codec>
                                                                        </BypassIfMatch>
                                                                </Audio>
                                                                <Audio>
                                                                        <Codec>opus</Codec>
                                                                        <Bitrate>128000</Bitrate>
                                                                        <Samplerate>48000</Samplerate>
                                                                        <BypassIfMatch>
                                                                                <Codec>eq</Codec>
                                                                        </BypassIfMatch>
                                                                </Audio>
                                                        </Encodes>
                                                </OutputProfile>

                                                <OutputProfile>
                                                        <Name>thumbnail</Name>
                                                        <OutputStreamName>${OriginStreamName}_preview</OutputStreamName>
                                                        <Encodes>
                                                                <Image>
                                                                        <Codec>jpeg</Codec>
                                                                        <Framerate>2</Framerate>
                                                                       <Width>192</Width>
                                                                        <Height>108</Height>
                                                                </Image>
                                                        </Encodes>
                                                </OutputProfile>

                                        </OutputProfiles>

[getroot]

Thank you for your thoughtful report. HEVC input support has been unmaintained due to low usage. (Because HEVC input is actually an experimental feature.)

I will soon start working on officially supporting AV1 and HEVC on OME. I'll most likely work on AV1 support first. (Because I saw an article saying that Apple finally started supporting AV1). If the AV1 input can be used as-is by WebRTC and LLHLS publishers without re-encoding, then there may not be much reason to use HEVC. Support for HEVC will be revisited later.

[bchah]

Thank you @getroot for the response. I understand that AV1 is starting to gain more support, and I am excited for that future too. But I look around at the dozens of encoders and computers I have here, knowing that their dedicated encoding hardware will only ever support H264 and HEVC (Like NVENC HEVC and Apple T2 chip). There are millions of streaming devices out there that would benefit from HEVC input support on OME, but will never be updated with the ability to create an AV1 stream.

In fact, HEVC support has been the top requested feature from customers since I started creating solutions around OME. The main reasons are:

1) The potential bandwidth savings. If a user has limited bandwidth for their server, the ability to receive and transcode HEVC to H264 (or AV1...) for viewers would allow them to either send twice the number of streams, or to improve the quality of the input stream at the same bitrate.

2) Additional colour depth and HDR support. HEVC has profiles that allow for 10-bit colour and HDR. While the WebRTC publisher would not be able to play them back directly without transcoding there is a lot of potential for use in LLHLS playback and recording functions.

So for these reasons I believe HEVC support is still very important. It will benefit all of the users with encoders today that could be using HEVC to save bandwidth or deliver a higher quality image. And it seems like with the SRT provider you are so close to having it working with HEVC... so I will keep hoping that one day you are bored and decide to look at it again 😅😅😅

As always thank you for your work in creating OME 🙏🏻

[getroot]

Yes, I also think the scenario of using HEVC in SRT -> OME -> LLHLS is very good. Having noticed that Enhanced RTMP (https://github.com/AirenSoft/OvenMediaEngine/discussions/1167) also supports HEVC, I'll prioritize this task more positively.

[naanlizard]

For our use case we only care that ome supports whatever obs supports, especially focusing on the most efficient encoder in terms of bandwidth usage (which we will recommend users use)

[bchah]

Thanks @naanlizard - I'm sure you'll agree that even in the use case of OBS being the only possible encoder, HEVC must still come up in conversation quite a bit. Looking at the encoder options on the current release for macOS and Windows confirms this. Wouldn't it be great to use them all? 😁😁😁

[getroot]

I fixed the HEVC SPS parsing error. (1d99c9db933b1122895042a7fbfc480b696dec9c)

OME's WHIP does not support HEVC. I will consider this in the future. (OBS's WHIP is still under development, so I'm not even sure if this works)

If HEVC video coming over SRT is encoded as AVC, it can be played in WebRTC and LLHLS.

I will support the HEVC codec in the LLHLS publisher in the future.

After testing, if you think this issue has been resolved, please close it.

Thanks for the report.

[bchah]

Thank you @getroot for enabling this feature! It works for receiving HEVC and transcoding to the H264 for the WebRTC provider 🎉🥳🍾

However I think I have discovered a memory leak - if you do not specify a Framerate option in the Video encode settings, transcoding to H264 does not succeed and starts to overwhelm the input queue. This continues until the OME utilizes all available RAM then restarts. This happens even if the input is H264, not just HEVC. I believe the intended behaviour is that the source framerate would be used if no Framerate option was specified.

This is how the input queue keeps building, along with the RAM usage. Let me know if you need any more info to track this down, thank you 👍

[2023-04-04 19:27:38.080] W [Rescaler:951] ov.Queue | queue.h:268  | [0x7f8849a01818] Input queue of Encoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 152, threshold: 120, peak: 152
[2023-04-04 19:27:43.121] W [Rescaler:951] ov.Queue | queue.h:268  | [0x7f8849a01818] Input queue of Encoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 303, threshold: 120, peak: 303
[2023-04-04 19:27:48.124] W [Rescaler:951] ov.Queue | queue.h:268  | [0x7f8849a01818] Input queue of Encoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 453, threshold: 120, peak: 453
[2023-04-04 19:27:53.132] W [Rescaler:951] ov.Queue | queue.h:268  | [0x7f8849a01818] Input queue of Encoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 602, threshold: 120, peak: 602
[2023-04-04 19:27:58.135] W [Rescaler:951] ov.Queue | queue.h:268  | [0x7f8849a01818] Input queue of Encoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 753, threshold: 120, peak: 753
[2023-04-04 19:28:30.162] W [Rescaler:951] ov.Queue | queue.h:268  | [0x7f8849a01818] Input queue of Encoder. track(0) codec(h264/27) size has exceeded the threshold: queue: 1122, threshold: 120, peak: 1122

[getroot]

@bchah Thanks for the great bug reporting. Could you please create a new issue?