[HOW-TO] Continuously stream lores, Occasionally record main

[schillingderek]

Describe what it is that you want to accomplish

Raspberry Pi Zero 2 W DietPi v9.6.1 - Debian GNU/Linux 12 (bookworm) Arducam OV5647

I am attempting to continuously stream the lores config, which I'm using for streaming video - and then occasionally capture the main config to a file.

The issue I'm having is that I can't understand how to get the streaming encoder to honor the size and format being passed in - it instead appears to just be using the main configuration - I can tell this because when the streaming video is decoded, it's clearly in the main size - changing the value of the lores size has no effect on the stream, but changing the main size most certainly does.

I have been trying to follow the example in dual_encode, but am not understanding how it works. Specifically this bit related to creating a request and then encoding the request:

    request = picam2.capture_request()
    mjpeg_encoder.encode(lores_stream, request)
    request.release()

When I read about the capture_request method in the docs, it seems like this is something that should only be done occasionally to avoid creating problems. But as I want to stream the lores config continuously, I'm not sure how to apply the idea to my situation.

If an application fails to release captured requests back to the camera system, the camera system will gradually run out of buffers. It
is likely to start dropping ever more camera frames, and eventually the camera system will stall completely.

I'm aware of the existing mjpeg_streaming_server example, and was trying to combine that with the idea of the dual_encode example to create high-quality video captures with a lower-quality stream. I found the MJPEG stream to be far too slow/choppy on my Pi Zero 2 W - but the H264 stream over websocket (based on this older Code Inside Out example, but updated to work with Picamera2) seems to work great.

Describe alternatives you've considered

This is my current attempt: https://github.com/schillingderek/birdCam/tree/main/streamingServer/stream_picamera_h264

The Picamera code in server.py I'll copy here for convenience:

#!/usr/bin/env python3

import io
import picamera2
from picamera2 import Picamera2
from picamera2.encoders import H264Encoder
from picamera2.outputs import FileOutput, CircularOutput
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
from wsgiref.simple_server import make_server
from ws4py.websocket import WebSocket
from ws4py.server.wsgirefserver import WSGIServer, WebSocketWSGIHandler, WebSocketWSGIRequestHandler
from ws4py.server.wsgiutils import WebSocketWSGIApplication
from threading import Thread, Condition
import time
from PIL import Image
from datetime import datetime

videoCaptureEncoder = H264Encoder()
videoCaptureOutput = CircularOutput()

startTime = time.time()

width = 960
height = 540

class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.frame = None
        self.condition = Condition()

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

    def read(self):
        with self.condition:
            self.condition.wait()
            return self.frame

class Camera:
    def __init__(self):
        self.camera = picamera2.Picamera2()
        self.video_config = self.camera.create_video_configuration({"size": (width, height)}, lores={"size": (640, 360)})
        self.camera.configure(self.video_config)
        self.streaming_encoder = H264Encoder()
        self.streaming_encoder.bitrate = 2500000
        self.streaming_encoder.profile = 'baseline'
        self.streaming_encoder.size = self.video_config['lores']['size']
        self.streaming_encoder.format = self.video_config['lores']['format']
        self.stream_out = StreamingOutput()
        self.stream_out_2 = FileOutput(self.stream_out)
        self.streaming_encoder.output = [self.stream_out_2]
        self.camera.start_encoder(self.streaming_encoder)

        self.camera.start_recording(videoCaptureEncoder, videoCaptureOutput)

        self.camera.start()

camera = Camera()

def stream():
    global startTime
    is_recording = False
    try:
        WebSocketWSGIHandler.http_version = '1.1'
        websocketd = make_server('', 9000, server_class=WSGIServer,
                 handler_class=WebSocketWSGIRequestHandler,
                 app=WebSocketWSGIApplication(handler_cls=WebSocket))
        websocketd.initialize_websockets_manager()
        websocketd_thread = Thread(target=websocketd.serve_forever)

        httpd = ThreadingHTTPServer(('', 8000), SimpleHTTPRequestHandler)
        httpd_thread = Thread(target=httpd.serve_forever)

        try:
            websocketd_thread.start()
            httpd_thread.start()

            while True:
                # Read from the StreamingOutput and broadcast via WebSocket
                frame_data = camera.stream_out.read()
                if frame_data:
                    #print("Sending frame of size:", len(frame_data))
                    websocketd.manager.broadcast(frame_data, binary=True)
                else:
                    print("No frame data received")
                # if time.time() - startTime > 10 and not is_recording:
                #     print("starting to record")
                #     timestamp = timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
                #     videoCaptureOutput.fileoutput = f"/root/birdcam/streamingServer/stream_picamera_h264/images/snap_{timestamp}.h264"
                #     videoCaptureOutput.start()
                #     is_recording = True
                #     startTime = time.time()
                # elif time.time() - startTime > 10 and is_recording:
                #     print("stopping recording")
                #     videoCaptureOutput.stop()
                #     is_recording = False
                #     startTime = time.time()

        except KeyboardInterrupt:
            pass
        finally:
            websocketd.shutdown()
            httpd.shutdown()
            camera.camera.stop()
            camera.camera.stop_encoder()
            raise KeyboardInterrupt
    except KeyboardInterrupt:
        pass
    finally:
        camera.camera.stop()
        camera.camera.stop_encoder()

if __name__ == "__main__":
    stream()

[davidplowman]

Hi, thanks for the question. I think the problem may just be that the easiest way to do this is to pass the encoder, output and stream name altogether to the start_encoder function. When you start the encoder, it should pick up the resolution, format and image stream from the name supplied - in your case it was probably overwriting whatever you'd tried to configure!

import time

from picamera2 import Picamera2
from picamera2.outputs import FileOutput
from picamera2.encoders import H264Encoder

with Picamera2() as picam2:
    main = {'size': (960, 540), 'format': 'YUV420'}
    lores = {'size': (640, 360), 'format': 'YUV420'}
    config = picam2.create_video_configuration(main, lores=lores)
    picam2.configure(config)

    streaming_encoder = H264Encoder()
    streaming_output = FileOutput("stream.h264")
    file_encoder = H264Encoder()
    file_output = FileOutput("file.h264")

    picam2.start()

    picam2.start_encoder(streaming_encoder, streaming_output, name='lores')
    time.sleep(2)
    picam2.start_encoder(file_encoder, file_output, name='main')
    time.sleep(5)
    picam2.stop_encoder(file_encoder)
    time.sleep(2)
    picam2.stop_encoder(streaming_encoder)

When you run this, you should get a ~5 second file at the larger resolution, and a ~9 second file at the lower resolution (you could play them back with ffplay).

[schillingderek]

Brilliant - works perfectly :) Thanks @davidplowman