decode example

[florisdesmedt]

Dear,

I would like to experiment with this library to replace my current approach to read streams (the OpenCV VideoCapture). However, I'm not completely familiar with how to decode the data I read (from a h264 stream) using PyAv. I stumble upon the error Error splitting the input into NAL units.

Do you have any example how to do the decoding?

[bdfy]

!/usr/bin/python3

` import asyncio from aiortsp.rtsp.reader import RTSPReader

from rtsp_rtp.transport.primitives import RTPDatagram, NalUnit from rtsp_rtp.transport.primitives.nal_unit import NalUnitError

f1 = open("H264-media-1.264", "wb")

async def main(): async with RTSPReader('rtspt://.......') as reader1: async for pkt in reader1.iter_packets(): rtp_payload = RTPDatagram(bytes(pkt)).payload nal_payload = NalUnit(rtp_payload).payload f1.write(nal_payload) f1.flush()

asyncio.run(main()) `

https://github.com/runtheops/rtsp-rtp.git

[RouquinBlanc]

Hi,

aiortsp aims at remaining as generic as possible and only deals with RTP. It is nice to see that there are other packages out there which can help processing the H264 Nals!

I will try to add some documentation

[davidmurray]

I managed to decode the RTP stream by piping the NAL payloads to ffmpeg and reading the decoded frames from it!

[RouquinBlanc]

If you get full NALs from the fragmented NALs you will receive via RTP, here is how to get the decoding done in PyAV:

import av

def decode(nal: bytes, ts: float, codec: av.CodecContext) -> Iterable[av.VideoFrame]:
    # Where the magic happens! That's THE line to remember.
    packet = av.packet.Packet(bytes([0, 0, 0, 1]) + data)
    packet.pts = ts * 1000  # ts in milliseconds
    return codec.decode(packet)

# Create a codec
codec = av.CodecContext.create('h264', 'r')

# Then for each frame you will have to call decode
frames = decode(NAL_data, NAL_ts, codec):

Note that before feeding any video NAL, you will have to first have PyAV decode the PPS and SPS nals.

[davidmurray]

Hi @RouquinBlanc, could you explain this line please? Why do we do this? Is it the NAL header?

packet = av.packet.Packet(bytes([0, 0, 0, 1]) + data)

Thanks!

[RouquinBlanc]

I don't know how to explain the 4 bytes to precede the NAL, that's just something I've been having to do both for:

• Feeding PyAV
• Creating MP4 out of H264 NALs with libmp4v2

It's a kind of preamble, but all the tooling around H264 and MP4 is a little obscure (to me at least). Just know that without this preamble, it does not work, or I missed something :-)

[berysaidi]

I hope this might help. playing the output file with ffplay works.

import asyncio
from aiortsp.rtsp.reader import RTSPReader

URL = 'rtsp://user:pw@ip:port/1/1'
OUTPUT_FILE = "cam_feed.h264"

'''
5.3.  NAL Unit Header Usage

   The structure and semantics of the NAL unit header were introduced in
   Section 1.3.  For convenience, the format of the NAL unit header is
   reprinted below:

      +---------------+
      |0|1|2|3|4|5|6|7|
      +-+-+-+-+-+-+-+-+
      |F|NRI|  Type   |
      +---------------+
'''
FNRI_MASK = 0xE0
NAL_TYPE_MASK = 0x1F
FU_START_BIT_MASK = 0x80
NAL_START= b'\x00\x00\x00\x01'

async def main():
    async with RTSPReader(URL) as reader:
        with open(OUTPUT_FILE, 'wb') as video_file:
            async for pkt in reader.iter_packets():
                nal_type = (pkt.data[0] & NAL_TYPE_MASK)
                '''
                https://datatracker.ietf.org/doc/html/rfc6184#section-5.3

                NAL Unit  Packet    Packet Type Name               Section
                Type      Type
                -------------------------------------------------------------
                0        reserved                                     -
                1-23     NAL unit  Single NAL unit packet             5.6
                24       STAP-A    Single-time aggregation packet     5.7.1
                25       STAP-B    Single-time aggregation packet     5.7.1
                26       MTAP16    Multi-time aggregation packet      5.7.2
                27       MTAP24    Multi-time aggregation packet      5.7.2
                28       FU-A      Fragmentation unit                 5.8
                29       FU-B      Fragmentation unit                 5.8
                30-31    reserved                                     -
                '''
                # Single NAL Unit Packets
                if nal_type in range(1, 24):
                    print("single NAL unit packet")
                    video_file.write(NAL_START)
                    video_file.write(pkt.data)
                # FU-A
                elif nal_type == 28:
                    print("FU-A packet")

                    '''
                    The FU header has the following format:

                          +---------------+
                          |0|1|2|3|4|5|6|7|
                          +-+-+-+-+-+-+-+-+
                          |S|E|R|  Type   |
                          +---------------+

                       S:     1 bit
                              When set to one, the Start bit indicates the start of a
                              fragmented NAL unit.  When the following FU payload is not the
                              start of a fragmented NAL unit payload, the Start bit is set
                              to zero.

                       E:     1 bit
                              When set to one, the End bit indicates the end of a fragmented
                              NAL unit, i.e., the last byte of the payload is also the last
                              byte of the fragmented NAL unit.  When the following FU
                              payload is not the last fragment of a fragmented NAL unit, the
                              End bit is set to zero.

                       R:     1 bit
                              The Reserved bit MUST be equal to 0 and MUST be ignored by the
                              receiver.

                       Type:  5 bits
                              The NAL unit payload type as defined in Table 7-1 of [1].

                       The value of DON in FU-Bs is selected as described in Section 5.5.

                          Informative note: The DON field in FU-Bs allows gateways to
                          fragment NAL units to FU-Bs without organizing the incoming NAL
                          units to the NAL unit decoding order.
                    '''
                    if (pkt.data[1] & FU_START_BIT_MASK):
                        print("FU-A packet start!")
                        video_file.write(NAL_START)
                        video_file.write(bytes([(pkt.data[0] & FNRI_MASK) | (pkt.data[1] & NAL_TYPE_MASK)]))
                    video_file.write(pkt.data[2:])
                # STAP-A, STAP-B
                elif nal_type in [24, 25]:
                    print("STRAP A/B packet")
                    # https://datatracker.ietf.org/doc/html/rfc6184#section-5.7.1
                    offset = 1
                    while offset < len(pkt.data):
                        nal_size = int.from_bytes(pkt.data[offset:offset+2], 'big')
                        offset += 2
                        video_file.write(NAL_START)
                        video_file.write(pkt.data[offset:offset+nal_size])
                        offset += nal_size
                elif nal_type in [26, 27]:  # MTAP16, MTAP24
                    print("MTAP16 MTAP24 packet (ignored)")
                    # Multi-time aggregation packet (MTAP): aggregates NAL units with
                    # potentially differing NALU-times.  Two different MTAPs are
                    # defined, differing in the length of the NAL unit timestamp offset.
                    # not sure what those are used for yet
                    pass

asyncio.run(main())