search

mpromonet / rtsp2web

RTSP to websocket proxy
https://rtsp2web.agreeabletree-365b9a90.canadacentral.azurecontainerapps.io/
The Unlicense
10 stars 1 forks source link

Question on h264 RTP packet decoding / forwarding to Websocket #4

Open antonioaguilar opened 2 weeks ago

antonioaguilar commented 2 weeks ago

@mpromonet Great work on implementing the rtsp2web server, the code is quite minimalist. I'm implementing a RTP/h264 relay server in Golang similar to yours.

I'm wondering if you might be able to clarify some questions on how the rtsp2web server is able to relay h264 media to the browser webcodecs:

1) is the server just relaying the RTP packets that contain the h264 media as-is to the Websocket connection? or,

2) is the server capturing the RTP packets, then decodes part or all of the h264 media into frames and then sends these decoded frames over the Websocket connection?

The EncodedVideoChunk() method of the Webcodecs API expects the data parameter to be an ArrayBuffer but doesn't specify if this data should be plain RTP packets or already decoded h264 media?

I'm asking all these because I'm not familiar (yet) how h264 or the Webcodecs API should work together.

mpromonet commented 2 weeks ago

Hi @antonioaguilar

The Rtp packets are extracted and transform to annexb format (see https://github.com/mpromonet/rtsp2web/blob/master/inc%2Frtspcallback.h), concatening SPS+PPS+IDR. This format could be used directly by VideoDecode.

I tried an implementaion in Rust https://github.com/mpromonet/rtsp2web-rs, it might helps, that convert avcc to annexb.

Best Regards Michel