How can I extract SPS, VPS from h.264 codec?

[eomgerm]

Hello. I'm developing live streaming application using OAK-1 Camera.

To stream video I need to extract SPS, VPS from video data. I successfully retrieved H264 data by referring to this article. When I access to NAL Unit Type I only get value 9 and I think this is not a normal situation.

However, like code below, I also saved the .h264 file and when I hex dumped it all NAL Unit Types are normal. Is my code wrong?

Please let me know how to extract SPS and VPS data using this library. Thanks

code I tried:

#!/usr/bin/env python3

import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define sources and output
camRgb = pipeline.create(dai.node.ColorCamera)
videoEnc = pipeline.create(dai.node.VideoEncoder)
xout = pipeline.create(dai.node.XLinkOut)

xout.setStreamName('h264')

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.AUTO)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
videoEnc.setDefaultProfilePreset(60, dai.VideoEncoderProperties.Profile.H264_MAIN)

# Linking
camRgb.video.link(videoEnc.input)
videoEnc.bitstream.link(xout.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    # Output queue will be used to get the encoded data from the output defined above
    q = device.getOutputQueue(name="h264", maxSize=30, blocking=True)

    # The .h265 file is a raw stream file (not playable yet)
    with open('video.h264', 'wb') as videoFile:
        print("Press Ctrl+C to stop encoding...")
        try:
            while True:
                h265Packet = q.get()  # Blocking call, will wait until a new data has arrived
                data = h265Packet.getData()
                print("NAL UNIT TYPE:", data[4] & 0x1F) # decoding NAL Unit Type
                h265Packet.getData().tofile(videoFile)  # Appends the packet data to the opened file
        except KeyboardInterrupt:
            # Keyboard interrupt (Ctrl + C) detected
            pass

    print("To view the encoded data, convert the stream file (.h264) into a "
          "video file (.mp4) using a command below:")
    print("ffmpeg -framerate 30 -i video.h264 -c copy video.mp4")

Output:

NAL UNIT TYPE: 9
NAL UNIT TYPE: 9
NAL UNIT TYPE: 9
NAL UNIT TYPE: 9
NAL UNIT TYPE: 9
... and so on.

[moratom]

@asahtik could you please give this one a look?

[asahtik]

Hi @eomgerm, in depthai an encoded frame message doesn't contain only a single NAL packet but contains many including SPS, PPS and slice packets, therefore you need to iterate over the packets in a message and extract the necessary data. You can refer to the methods in H26xParsers.cpp which find slice NAL units and extract the frame type (the NAL unit type that you're interested in is extracted in line 171).