Try h265

[mpromonet]

Description

This pull request try to use recently support h265 parsing in webrtc SDK, with -o it forward h265 packet from RTSP to WebRTC. It needs a web browser that support webrtc with h265 to validate and improve it.

Related Issue

https://github.com/mpromonet/webrtc-streamer/issues/600 https://github.com/mpromonet/webrtc-streamer/issues/611

Motivation and Context

Types of changes

• [ ] Docs change / refactoring / dependency upgrade
• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

[okteto-cloud[bot]]

Preview environment mpromonet-webrtc-streamer-pr-615 is being deployed...

[liusuyi2021]

谷歌已经支持HEVC的H265了，使用最新版本0.8.2 还是无法播放呢？

[yangjieshao]

所以 现在已经支持H265了？ (So now it supports H265?) congratulations 有人可以提供一下支持的浏览器版本号么？ (Can someone provide a supported browser version?)

[yangjieshao]

是否需要更新 webrtcstreamer.js (is that need to update webrtcstreamer.js？)

[yangjieshao]

可能是我chrome版本不对，画面并未在浏览器显示 (Perhaps my Chrome version is incorrect and the video is not displayed in the browser)

[yangjieshao]

看上去是要更新 webrtcstreamer.js ？ (t seems like we need to update webrtcstreamer.js?)

[mpromonet]

Hi,

The H265 support is not limited by webrtcstreamer.js, you can use another way to display webrtc stream it will be the same. You can print video codec supported by webrtc stack of your browser with :

console.log(RTCRtpReceiver.getCapabilities('video').codecs)

Best Regards, Michel.

[strong0716]

I used the same JavaScript code to set the RTSP URL, and updated the webrtc-streamer executable to the latest version (v0.8.3). I utilized the "-o" option in the command line to pass the stream to the web. When the stream is in h264 format, I can view the video in the browser; however, I am unable to do so when the stream is in h265 format. My JavaScript code involves importing "webrtcstreamer.js" for use. T_T

(ps.) my chrome version is 118.0.5993.71

[mpromonet]

Hi @strong0716,

I am sorry to do the same answer than before, but it is needed that the browser support h265, you can check this with console.log(RTCRtpReceiver.getCapabilities('video').codecs). You may also read this https://github.com/AlexxIT/Blog/issues/5

Best Regards, Michel.

[strong0716]

I tested the command console.log(RTCRtpReceiver.getCapabilities('video').codecs) in my Chrome browser and indeed, I didn't see support for h265. Initially, I thought that the Chrome browser had announced support for h265, but it appears that the support is for general video streaming. It seems that webrtc does not have support for h265 yet.

Thank you!

[LucaArteco]

I've checked with the Tech Preview on Safari (Mac) (Release 182 (Safari 17.4, WebKit 19618.1.4.1)) and from the capabilities call should be supported but I didn't get it working neither.

Basically I'm not getting errors, but checking the decoded frames

pc.getStats().then(res => {
    res.forEach(report => {
    let bytes;
    let headerBytes;
    let packets;
    if (report.type === 'inbound-rtp') {
        if(report.framesDecoded > 0 && !firstFrame) {                                   
            firstFrame = true;

... framesDecoded is 0. What else should I try? Many thanks! Luca

[mpromonet]

Hi @LucaArteco,

If you are using -o option, the H265 frame should be forwarded through webrtc. Is it working with a H264 stream ?

Best Regards, Michel.

[LucaArteco]

Hi @mpromonet yes everything's fine on H264, and I kept the -o option.

Recap:

• WebRTC H.264: Working
• Tested my RTSP out with VLC: Working
• RTSP out video info: confirmed HEVC Codec
• RTCRtpReceiver.getCapabilities('video').codecs: Confirmed presence video/H.265 in list (ONLY current Safari Tech Prev)
• WebRTC HEVC: No errors, but no decoded frame (the screen is just black)

Did someone actually see the HEVC WebRTC stream working on any browser? I would like to replicate a working example to see if maybe the issues are in my RTSP out (VLC can play it but VLC eats everything usually)

EDIT: Just adding some logs after comparing two sessions. One working H.264 and one not working (H.265)

I made a little amendment in case of H.265 in order to force the priority for that code. It worked (for what I got) and now in case of h265 source I got the H265 (104 in my SDP) as first codec for both Local and Remote Description.

webrtc-streamer log H.264 Start playing sink for "video/H264" subsession H.265 Start playing sink for "video/H265" subsession

[247:283][14236] (PeerConnectionManager.h:183): OnStateChange channel:ServerDataChannel state:open
[247:284][14236] (PeerConnectionManager.h:274): OnDataChannel
[247:286][14236] (PeerConnectionManager.h:190): OnMessage channel:ServerDataChannel msg:remote channel openned
[247:286][14236] (PeerConnectionManager.h:183): OnStateChange channel:ClientDataChannel state:open
[247:287][14236] (PeerConnectionManager.h:190): OnMessage channel:ClientDataChannel msg:local channel openned

Codec stats (from pc.getStats()) H.265

clockRate: 90000
id: "COT01_104"
mimeType: "video/H265"
payloadType: 104
timestamp: 1699632833081
transportId: "T01"
type: "codec"

H.264

clockRate: 90000
id: "COT01_98_level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f"
mimeType: "video/H264"
payloadType: 98
sdpFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f"
timestamp: 1699633000524
transportId: "T01"
type: "codec"

I've noticed that additional parameter, sdpFmtpLine, which is not appearing for H265

Thanks! Luca

[mpromonet]

Hi @LucaArteco,

I am still waiting for a support on windows or linux to make tests... Maybe parsing https://github.com/mpromonet/webrtc-streamer/blob/master/inc/VideoDecoder.h#L74C1-L75C1 is not working well ? Increasing loglevel may helps to understand problem. This commit https://github.com/mpromonet/webrtc-streamer/commit/d0bef6c4872baec79b02ad3c72f0fa295a9c55f2 may helps too.

Best Regards, Michel.

[LucaArteco]

Hi @mpromonet thanks, we're now testing your commit. What kind of support are you waiting for? If you need test cases with different platform and camera brands we're available, feel free to propose. Now we're checking your suggestions and let you know. Many thanks, Luca

[mpromonet]

Hi Luca, The problem is to identify where it is stop working... maybe server doesnot send the webrtc stream, maybe browser doesnot support what it receive.... Enabling server log may helps to identify if negociation works well and send data.... Are you able to negociate H265 webrtc stream with another tools, like between 2 webbrowsers ? go2rtc ? Best Regards, Michel.

[LucaArteco]

Hi Michel following your refs, i've stumbled upon this PoC https://github.com/jimm98y/SharpRTSPtoWebRTC and I've been able to play H265 on Safari (17.4, also not tech preview).

I think the key reference is still https://github.com/AlexxIT/Blog/issues/5

I've tested your last commit but the problem is still I'm receiving bytes, but no frames were decoded on client side. On server side I got these logs

[152:668][3528] (h265_sps_parser.cc:516): Error in stream: invalid value, expected sps_max_num_reorder_pics[i] to be in range [0:0] found 3 instead
[152:668][3528] (livevideosource.h:192): cannot parse sps
[152:698][1420] (video_stream_encoder.cc:1774): Video frame parameters changed: dimensions=0x0, texture=1.
[152:698][1420] (VideoEncoderFactory.h:26): Create Null Encoder format:Codec name: H265, parameters: { }
[152:699][1420] (video_stream_encoder.cc:1196): ReconfigureEncoder:
Simulcast streams:
0: 0x0 min_kbps: 30 target_kbps: 600 max_kbps: 600 max_fps: 60 max_qp: 51 num_tl: 1 active: true

[152:700][1420] (NullEncoder.h:24): InitEncode
[152:701][1420] (resource_adaptation_processor.cc:117): Registered resource "EncoderUsageResource".
[152:701][1420] (quality_scaling_experiment.cc:41): QP thresholds: low: 24, high: 37
[152:701][1420] (quality_scaler.cc:216): QP thresholds: low: 24, high: 37
[152:702][1420] (resource_adaptation_processor.cc:117): Registered resource "QualityScalerResource".

On client side, I see Bytes flowing but no decoding

Many thanks, Luca

[mpromonet]

Hi Luca,

In the log there is an error parsing SPS, but there is also an error forwarding PPS, I fixed this in https://github.com/mpromonet/webrtc-streamer/commit/093e8b2111235308b2139fe0b282f2879ab5c2a3.

Best Regards, Michel.

[LucaArteco]

Hi Michel, still getting the sps parsing error

Best, Luca

[mpromonet]

Hi luca, Sure, the fix allow to forward the pps that was broken, parsing SPS is the same. Maybe webrtc code doesnot support camera SPS, maybe a problem in https://webrtc.googlesource.com/src/+/refs/heads/main/common_video/h265/h265_sps_parser.cc#516. Not sure the safari annexb stuff is implemented in webrtc... Best Regards Michel

[jimm98y]

Safari 17.5 no longer uses H265 AnnexB packetization, they switched to RFC 7798 which means almost the same code as for H264 should also work for H265.