search

raspberrypi / picamera2

New libcamera based python library
BSD 2-Clause "Simplified" License
890 stars 188 forks source link

[BUG] Cannot stream MJPEG #788

Open stuartofmt opened 1 year ago

stuartofmt commented 1 year ago

Describe the bug Testing streaming of USB camera. Works with Pi camera but not USB. I realize that full support for USB may not be available, but it seems this is a straightforward use case that should work.

USB camera displays stills in preview with picam.create_preview_configuration({"format": "MJPEG"})

Pi camera works fine with: picam2.configure(picam2.create_video_configuration()) but not
picam2.configure(picam2.create_video_configuration({"format": "MJPEG"}))

USB camera throws an error: ValueError: Invalid format: MJPEG. Valid formats are: ...

This occures when attempting to capture video with either: picam2.configure(picam2.create_video_configuration()) or picam2.configure(picam2.create_video_configuration({"format": "MJPEG"}))

I tried YUYV (a valid format and reported by camera as available)
picam2.configure(picam2.create_video_configuration({"format": "YUYV"})) This gave a different error (see below)

To Reproduce This following sample code is taken from one of the examples with some added code to provide info about the cameras. All changes are at the end of the code.

#!/usr/bin/python3

# Mostly copied from https://picamera.readthedocs.io/en/release-1.13/recipes2.html
# Run this script, then point a web browser at http:<this-ip-address>:8000
# Note: needs simplejpeg to be installed (pip3 install simplejpeg).

import io
import logging
import socketserver
from http import server
from threading import Condition

from picamera2 import Picamera2
from picamera2.encoders import JpegEncoder
from picamera2.outputs import FileOutput

PAGE = """\
<html>
<head>
<title>picamera2 MJPEG streaming demo</title>
</head>
<body>
<h1>Picamera2 MJPEG Streaming Demo</h1>
<img src="stream.mjpg" width="640" height="480" />
</body>
</html>
"""

class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.frame = None
        self.condition = Condition()

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

class StreamingHandler(server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/':
            self.send_response(301)
            self.send_header('Location', '/index.html')
            self.end_headers()
        elif self.path == '/index.html':
            content = PAGE.encode('utf-8')
            self.send_response(200)
            self.send_header('Content-Type', 'text/html')
            self.send_header('Content-Length', len(content))
            self.end_headers()
            self.wfile.write(content)
        elif self.path == '/stream.mjpg':
            self.send_response(200)
            self.send_header('Age', 0)
            self.send_header('Cache-Control', 'no-cache, private')
            self.send_header('Pragma', 'no-cache')
            self.send_header('Content-Type', 'multipart/x-mixed-replace; boundary=FRAME')
            self.end_headers()
            try:
                while True:
                    with output.condition:
                        output.condition.wait()
                        frame = output.frame
                    self.wfile.write(b'--FRAME\r\n')
                    self.send_header('Content-Type', 'image/jpeg')
                    self.send_header('Content-Length', len(frame))
                    self.end_headers()
                    self.wfile.write(frame)
                    self.wfile.write(b'\r\n')
            except Exception as e:
                logging.warning(
                    'Removed streaming client %s: %s',
                    self.client_address, str(e))
        else:
            self.send_error(404)
            self.end_headers()

class StreamingServer(socketserver.ThreadingMixIn, server.HTTPServer):
    allow_reuse_address = True
    daemon_threads = True

###  Starts here
available_cameras = Picamera2.global_camera_info()

for cam in range(len(available_cameras)):
    print(f'---- Details for camera {cam}')
    picam2 = Picamera2(cam)
    print(f'Camera: {available_cameras[cam]["Model"]}')
    print(picam2.sensor_modes)
    print(picam2.sensor_resolution)
    print(picam2.create_video_configuration())
    picam2.close()
    print('\n\n\n')

picam2 = Picamera2(1)
# picam2.configure(picam2.create_video_configuration())
picam2.configure(picam2.create_video_configuration({"format": "MJPEG"}))
output = StreamingOutput()
picam2.start_recording(JpegEncoder(), FileOutput(output))

try:
    address = ('', 8081)
    server = StreamingServer(address, StreamingHandler)
    server.serve_forever()
finally:
    picam2.stop_recording()

Expected behaviour That it works :-) or maybe a workaround?

Console Output, Screenshots I added a few spaces to the console output for ease of reading.

I notice that the sensor.modes information from the pi camera (lots of it) is very different to that of the USB camera([{'format': 'MJPEG'}, {'format': 'YUYV'}]).

The output using picam2.configure(picam2.create_video_configuration()) or picam2.configure(picam2.create_video_configuration({"format": "MJPEG"}))

[43:40:01.815888245] [7540]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[43:40:01.891635938] [7541]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[43:40:01.893642427] [7541]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[43:40:01.894019715] [7541]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'

---- Details for camera 0
[43:40:01.974919645] [7540]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[43:40:02.047390862] [7544]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[43:40:02.049173812] [7544]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[43:40:02.049286571] [7544]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'

Camera: imx219

[43:40:02.067818512] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 640x480-SBGGR10_CSI2P
[43:40:02.068869751] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 640x480-SBGGR10_1X10 - Selected unicam format: 640x480-pBAA
[43:40:02.092751166] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 1640x1232-SBGGR10_CSI2P
[43:40:02.093790374] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected unicam format: 1640x1232-pBAA
[43:40:02.130956077] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 1920x1080-SBGGR10_CSI2P
[43:40:02.132345645] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
[43:40:02.186451899] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 3280x2464-SBGGR10_CSI2P
[43:40:02.187829697] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 3280x2464-SBGGR10_1X10 - Selected unicam format: 3280x2464-pBAA
[43:40:02.261862148] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 640x480-SBGGR8
[43:40:02.276421161] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 640x480-SBGGR8_1X8 - Selected unicam format: 640x480-BA81
[43:40:02.334499977] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 1640x1232-SBGGR8
[43:40:02.335554237] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR8_1X8 - Selected unicam format: 1640x1232-BA81
[43:40:02.380179080] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 1920x1080-SBGGR8
[43:40:02.381506306] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR8_1X8 - Selected unicam format: 1920x1080-BA81
[43:40:02.444123668] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 640x480-XBGR8888 (1) 3280x2464-SBGGR8
[43:40:02.445836931] [7544]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 3280x2464-SBGGR8_1X8 - Selected unicam format: 3280x2464-BA81
[{'format': SRGGB10_CSI2P, 'unpacked': 'SRGGB10', 'bit_depth': 10, 'size': (640, 480), 'fps': 103.33, 'crop_limits': (1000, 752, 1280, 960), 'exposure_limits': (75, None)}, {'format': SRGGB10_CSI2P, 'unpacked': 'SRGGB10', 'bit_depth': 10, 'size': (1640, 1232), 'fps': 41.85, 'crop_limits': (0, 0, 3280, 2464), 'exposure_limits': (75, 11766829, None)}, {'format': SRGGB10_CSI2P, 'unpacked': 'SRGGB10', 'bit_depth': 10, 'size': (1920, 1080), 'fps': 47.57, 'crop_limits': (680, 692, 1920, 1080), 'exposure_limits': (75, 11766829, None)}, {'format': SRGGB10_CSI2P, 'unpacked': 'SRGGB10', 'bit_depth': 10, 'size': (3280, 2464), 'fps': 21.19, 'crop_limits': (0, 0, 3280, 2464), 'exposure_limits': (75, 11766829, None)}, {'format': SRGGB8, 'unpacked': 'SRGGB8', 'bit_depth': 8, 'size': (640, 480), 'fps': 103.33, 'crop_limits': (1000, 752, 1280, 960), 'exposure_limits': (75, 11766829, None)}, {'format': SRGGB8, 'unpacked': 'SRGGB8', 'bit_depth': 8, 'size': (1640, 1232), 'fps': 41.85, 'crop_limits': (0, 0, 3280, 2464), 'exposure_limits': (75, 11766829, None)}, {'format': SRGGB8, 'unpacked': 'SRGGB8', 'bit_depth': 8, 'size': (1920, 1080), 'fps': 47.57, 'crop_limits': (680, 692, 1920, 1080), 'exposure_limits': (75, 11766829, None)}, {'format': SRGGB8, 'unpacked': 'SRGGB8', 'bit_depth': 8, 'size': (3280, 2464), 'fps': 21.19, 'crop_limits': (0, 0, 3280, 2464), 'exposure_limits': (75, 11766829, None)}]

(3280, 2464)

{'use_case': 'video', 'transform': <libcamera.Transform 'identity'>, 'colour_space': <libcamera.ColorSpace 'Rec709'>, 'buffer_count': 6, 'queue': True, 'main': {'format': 'XBGR8888', 'size': (1280, 720)}, 'lores': None, 'raw': {'format': 'SBGGR10_CSI2P', 'size': (1280, 720)}, 'controls': {'NoiseReductionMode': <NoiseReductionModeEnum.Fast: 1>, 'FrameDurationLimits': (33333, 33333)}, 'display': 'main', 'encode': 'main'}

---- Details for camera 1
[43:40:02.739877953] [7540]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[43:40:02.834190398] [7548]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[43:40:02.836145065] [7548]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[43:40:02.836309438] [7548]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'

Camera: Microsoft LifeCam HD-3000: Mi

[{'format': 'MJPEG'}, {'format': 'YUYV'}]

(1280, 720)

{'use_case': 'video', 'transform': <libcamera.Transform 'identity'>, 'colour_space': <libcamera.ColorSpace 'Rec709'>, 'buffer_count': 6, 'queue': True, 'main': {'format': 'XBGR8888', 'size': (1280, 720)}, 'lores': None, 'raw': None, 'controls': {}, 'display': 'main', 'encode': 'main'}

[43:40:03.056194782] [7540]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[43:40:03.130713790] [7552]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[43:40:03.132523094] [7552]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[43:40:03.132719551] [7552]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[43:40:03.159512706] [7540]  INFO Camera camera.cpp:1033 configuring streams: (0) 1280x720-MJPEG
Traceback (most recent call last):
  File "/home/pi/videostream/./videostream-lite.py", line 103, in <module>
    picam2.start_recording(JpegEncoder(), FileOutput(output))
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1597, in start_recording
    self.start_encoder(encoder, output, pts=pts, quality=quality, name=name)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1518, in start_encoder
    _encoder.format = streams[name]['format']
  File "/usr/lib/python3/dist-packages/picamera2/encoders/encoder.py", line 156, in format
    formats.assert_format_valid(value)
  File "/usr/lib/python3/dist-packages/picamera2/formats.py", line 39, in assert_format_valid
    raise ValueError(f"Invalid format: {fmt}. Valid formats are: {ALL_FORMATS}")
ValueError: Invalid format: MJPEG. Valid formats are: {'NV21', 'YUYV', 'R12', 'RGB888', 'R10', 'R8_CSI2P', 'SRGGB10', 'SGBRG12', 'SGRBG8', 'SGBRG12_CSI2P', 'SGBRG10_CSI2P', 'SGRBG12_CSI2P', 'R12_CSI2P', 'SRGGB12_CSI2P', 'NV12', 'BGR888', 'SGRBG12', 'SGBRG8', 'SRGGB12', 'SGBRG10', 'SRGGB10_CSI2P', 'UYVY', 'VYUY', 'YVYU', 'SBGGR12_CSI2P', 'XRGB8888', 'R8', 'YVU420', 'XBGR8888', 'SGRBG10', 'SBGGR10', 'SBGGR10_CSI2P', 'SGRBG10_CSI2P', 'SBGGR12', 'SRGGB8', 'SBGGR8', 'R10_CSI2P', 'YUV420'}

The error using picam2.configure(picam2.create_video_configuration({"format": "YUYV"}))

[44:35:30.213029403] [8098]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[44:35:30.310570269] [8110]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[44:35:30.312305189] [8110]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[44:35:30.312428416] [8110]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[44:35:30.337001871] [8098]  INFO Camera camera.cpp:1033 configuring streams: (0) 1280x720-YUYV
Traceback (most recent call last):
  File "/home/pi/videostream/./videostream-lite.py", line 104, in <module>
    picam2.start_recording(JpegEncoder(), FileOutput(output))
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1597, in start_recording
    self.start_encoder(encoder, output, pts=pts, quality=quality, name=name)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1522, in start_encoder
    min_frame_duration = self.camera_ctrl_info["FrameDurationLimits"][1].min
KeyError: 'FrameDurationLimits'

Hardware : Pi 3B+ Glamor enabled Bullseye 64

stuartofmt commented 1 year ago

As a follow up ...

Based in this part of the error when using format : YUYV KeyError: 'FrameDurationLimits'

I tried

config = picam2.create_video_configuration(main={"format" : "YUYV"},controls={'FrameRate': 25})

and

config = picam2.create_video_configuration(main={"format" : "YUYV"},controls={'FrameDurationLimits': (40000 , 40000)})

which resulted in:

Traceback (most recent call last):
  File "/home/pi/videostream/./videostream-lite.py", line 106, in <module>
    picam2.configure(config)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1004, in configure
    self.configure_(camera_config)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 999, in configure_
    self.controls = Controls(self, controls=self.camera_config['controls'])
  File "/usr/lib/python3/dist-packages/picamera2/controls.py", line 24, in __init__
    self.set_controls(controls)
  File "/usr/lib/python3/dist-packages/picamera2/controls.py", line 58, in set_controls
    self.__setattr__(k, v)
  File "/usr/lib/python3/dist-packages/picamera2/controls.py", line 33, in __setattr__
    raise RuntimeError(f"Control {name} is not advertised by libcamera")
RuntimeError: Control FrameDurationLimits is not advertised by libcamera
davidplowman commented 1 year ago

Hi, I'm afraid support for USB cameras is very limited, and there are no plans to extend support for them.

stuartofmt commented 1 year ago

Understood. To what extent is there support for USB cameras?

I'm asking only so I don't go down a rabbit hole of investigation. I can (and have) handled USB cameras with openCV but was hoping to avoid that due to the various problems getting openCV to install on pi's of different stripes.

In other words - even experienced folks may have to try several techniques to get openCV installed. My program is aimed at inexperienced folks :-)

Probably what I'll do is create a "lite" version just for dedicated pi cameras.

davidplowman commented 1 year ago

I think the support extends to getting frames out of USB cameras, and displaying them in the slow (non-EGL accelerated) preview. I think you can save it as a JPEG as well (which obviously doesn't need to do an extra JPEG encode now). I use USB cams mostly for testing multi-camera functionality without requiring a compute module!

You can't do "video encode" so far as I know, though there's a chance you could make something work by tossing the JPEG frames at the base Encoder class, for which encode is a no-op. I suppose that might even allow you to stream them, though I don't think USB cams are sufficiently important to us to devote resources to that.

I recommend installing OpenCV from apt (sudo apt install python3-opencv, I think). I've had no end of trouble installing from other sources, where it goes off and spends a day compiling stuff and then failing at the final hurdle. I would install Qt from apt as well, otherwise it can assume a different version of OpenCV from what you have (if you didn't get OpenCV from apt). It's all a bit of a minefield.