[HOW-TO] Parse raw recorded footage

[SidSidSid16]

I am recording raw footage from a global shutter camera. The footage size is 840x640 and I intend to record in the SBGGR10 format.

import cv2 as cv
import numpy as np
from picamera2 import Picamera2
from picamera2.encoders import Encoder
from datetime import datetime

# Width and height of the recording
REC_WIDTH = 840
REC_HEIGHT = 640

# initialise picam2
picam = Picamera2()

# configure picam2
picam.video_configuration.enable_raw()
picam.video_configuration.raw.size = (REC_WIDTH, REC_HEIGHT)
picam.video_configuration.raw.format = 'SBGGR10'
picam.video_configuration.size = (REC_WIDTH, REC_HEIGHT)
picam.configure("video")

# window to display recording feed (view from X11)
cv.namedWindow("PiCam Feed")

recording = False

encoder = Encoder()

# start the picam
picam.start()

while True:
    # capture a frame from the picam
    frame = picam.capture_array()
    # clone to overlay text
    frame_overlay = frame.copy()

    # detect a key press
    key = cv.waitKey(1)

    # exit if 'e' key is pressed
    if key == ord('e'):
        # refuse to exit if a video is being recorded
        if recording:
            frame_overlay = cv.putText(
                img=frame_overlay,
                text="     CANNOT EXIT: REC IN PROGRESS!",
                org=(10, 50),
                fontFace=cv.FONT_HERSHEY_SIMPLEX,
                fontScale=1,
                color=(0,0,255),
                thickness=2,
                lineType=cv.LINE_AA
            )
        else:
            # stop the picam
            picam.stop()
            # break out of while loop
            break

    # toggle recording if 'r' pressed
    if key == ord('r'):
        if recording:
            recording = False
            picam.stop_recording()
            print("Recording finished")
            picam.stop()
            picam.start()
        else:
            recording = True
            recording_start = datetime.now().strftime("%m-%d-%Y-%H-%M-%S")
            filename = "recording_" + recording_start + ".raw"
            print("Started recording to:", filename)
            # use picam recording function with the right encoder
            picam.start_recording(
                encoder=encoder,
                output=filename,
            )

    # add text overlay to gui frame if recording
    if recording:
        frame_overlay = cv.putText(
            img=frame_overlay,
            text="REC",
            org=(10, 50),
            fontFace=cv.FONT_HERSHEY_SIMPLEX,
            fontScale=1,
            color=(255,0,0),
            thickness=2,
            lineType=cv.LINE_AA
        )

    # show the GUI frame
    cv.imshow("PiCam Feed", frame_overlay)

# when exiting, destroy all GUI windows
cv.destroyAllWindows()

How can I go about parsing this to play back the recorded footage?

[davidplowman]

Hi, from your question I understand that you want to save the raw Bayer data that came from the sensor, is that correct? The first thing is that you probably need to do is tell the encoder to use the "raw" stream (it defaults to the "main" one). Use

picam.video_configuration.encode = 'raw'

to do this. Thereafter, the raw data buffers should be dumped to disk with no formatting at all. If you look at picam2.video_configuration after configuring the camera, this should report the size and stride value of the raw images. You're asking for an unpacked raw format which means every pixel should occupy exactly 2 bytes.

One of the challenges you will have may be finding a disk that will keep up!

[SidSidSid16]

Thank you very much for your response!

I have updated the configuration part of my script to this:

picam.video_configuration.enable_raw()
picam.video_configuration.raw.size = (REC_WIDTH, REC_HEIGHT)
picam.video_configuration.raw.format = 'SBGGR10'
picam.video_configuration.size = (REC_WIDTH, REC_HEIGHT)
picam.video_configuration.align()
picam.video_configuration.encode = 'raw'
print(picam.video_configuration)
picam.configure("video")
print(picam.video_configuration)

I've noticed a slightly odd behaviour where the REC_WIDTH and REC_HEIGHT that I set are reset back to the default after picam.configure("video"). Am I missing something and is this the intended behaviour of the library? Is this because the raw stream bypasses the image signal processor so resizing isn't compatible?

Also, are there any resources that I can look at to help parse and playback the raw Bayer SBGGR10 video file?

[davidplowman]

You certainly have to be a bit careful with raw streams. Because they come straight from the sensor, there are only very limited resolutions available (type libcamera-hello --list-cameras into a console window to see what they are). Moreover, if you really want the raw stream then you may as well ask for a fairly small main stream just to cut down on memory traffic.

I don't really know of any good resources as regards using that raw data, it all depends what you want to do with it. Turning it into proper pictures is a relatively involved and expensive task. But you should find that you have stride x height bytes per frame (stride bytes per image row), representing a width x height image, with every pixel occupying 2 bytes.

Often folks like to save raw buffers as DNG files. The Picamera2 recording feature doesn't do this automatically, but I recall a discussion where this was implemented, here: https://forums.raspberrypi.com/viewtopic.php?p=2174531#p2161982 . As I mention in that thread, disk I/O speed can be a real problem.

[SidSidSid16]

Thank you very much for the help. I did explore this matter further, and sure enough, as you mentioned, I ran into too many problems regarding disk I/O speeds, even using a memory buffer I wasn't able to record footage long enough, and, of course, the challenges of turning it into a proper picture.

In the end, I used the 'main' stream as you suggested, with controls to the frame rate and resolution in XBGR8888 format, I recorded the processed raw footage without any encoding. I was able to generate a 'preview' window using capture_array("main") and feeding the output directly to OpenCV via cv2.imshow() in a while True: loop.

I used the picamera2 start_recording() and stop_recording() functions, outputting to BytesIO and writing to a file after the recording stopped. I haven't yet started writing the viewing script to parse the recording file for this and generate a video playback. But I will use your help with the stride and height values to parse, luckily as it's already processed by the Pi's Image Signal Processor, the output will be human readable.

I'm happy for this issue to be closed if it's ok for me to open another if I run into any issues along the way. Thank you.