take pictures during a video recording -errors

I want to take pictures during a video recording (webstream) is in process.

I basically follow the still_during_video.py example:

Describe the bug

Reconfigure (to video mode) the camera after picture has been taken failed. (see below)
With the reconfiguration:
Several other errors like "MmemoryAllocation" which I will describe later after I am sure that in my code are no bug itsself.

To Reproduce Because of #887 I have also tested in the next branch.

#!/usr/bin/python3

from typing import Final
from pathlib import Path
import time
import io
from threading import Condition

from picamera2.encoders import MJPEGEncoder
import libcamera
from picamera2.picamera2 import Picamera2
from picamera2.outputs import FileOutput

class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.__frame = None
        self.__condition = Condition()

    @property
    def frame(self) -> bytes:
        return self.__frame

    @frame.setter
    def frame(self, frame: bytes) -> None:
        self.__frame = frame

    @property
    def condition(self) -> Condition:
        return self.__condition

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

STILL_CONFIGURATION: Final  = {
    'use_case': 'still',
    'transform': libcamera.Transform(hflip=1),
    'colour_space': libcamera.ColorSpace.Sycc(),
    'buffer_count': 1,
    'queue': True,
    'main': {'format': 'BGR888', 'size': (1920, 1080)},
    'lores': None,
    'raw': {'format': 'SRGGB12_CSI2P', 'size': (4056, 3040)},
    'controls': {
                 'NoiseReductionMode': libcamera.controls
                 .draft.NoiseReductionModeEnum.HighQuality,
                 'FrameDurationLimits': (100, 1000000000),
                 'Saturation': 0.0,
                 'Sharpness': 12.0,
                 'Brightness': 0.0,
                 'AnalogueGain': 5,
                 'ColourGains': (0, 0),
                 'AeExposureMode': libcamera.controls
                 .AeExposureModeEnum.Normal,
                 'AwbEnable': False,
                 'AeEnable': False,
                 'ExposureTime': 10000},
    'sensor': {'output_size': (4056, 3040)},
    'display': None, 'encode': None}

picam2 = Picamera2()

def start_recording() -> None:
    encoder = MJPEGEncoder()
    picam2.start_recording(
        encoder=encoder,
        output=FileOutput(StreamingOutput())) 

def take_picture():
    request = None
    if picam2.started:
        request = picam2.capture_request()
    started = picam2.started
    config = picam2.camera_config  #config = copy.deepcopy(picam2.camera_config)
    picam2.stop()
    picam2.configure(STILL_CONFIGURATION)
    picam2.start()
    img = picam2.capture_image("main")
    picam2.stop()
    picam2.configure(config)
    if request is not None:
         request.release()
    if started:
        picam2.start()

def test_loop():
    start_recording()
    i = 0
    while True:
        print("\n \n \n ------------------", i)
        take_picture()
        time.sleep(1.0)
        i += 1
        if i > 5:
            break

if __name__ == '__main__':
    test_loop()

Expected behaviour Picture taking works during video recording running infinity number of times.

Console Output, Screenshots 1.

[10:53:49.657379752] [8609]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x720-XBGR8888 (1) 2028x1080-SBGGR12_CSI2P
[10:53:49.669070213] [8604]  INFO RPI vc4.cpp:608 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1080-SBGGR12_1X12 - Selected unicam format: 2028x1080-pBCC
Exception in thread Thread-2 (thread_poll):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/picamera2/encoders/v4l2_encoder.py", line 241, in thread_poll
    queue_item.release()
  File "/usr/lib/python3/dist-packages/picamera2/request.py", line 140, in release
    self.picam2.allocator.release(self.request.buffers)
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 78, in release
    self.cleanup()
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 81, in cleanup
    for k, v in self.mapped_buffers.items():
RuntimeError: dictionary changed size during iteration
G
Traceback (most recent call last):
  File "/home/pi/camera_websetter_python/./test_loop.py", line 116, in <module>
    test_loop()
  File "/home/pi/camera_websetter_python/./test_loop.py", line 105, in test_loop
    take_picture()
  File "/home/pi/camera_websetter_python/./test_loop.py", line 95, in take_picture
    request.release()
  File "/usr/lib/python3/dist-packages/picamera2/request.py", line 140, in release
    self.picam2.allocator.release(self.request.buffers)
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 78, in release
    self.cleanup()
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 83, in cleanup
    if not self.mapped_buffers_used[fd] and fd not in self.libcamera_fds:
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^
KeyError: 27
Exception ignored in: <function DmaAllocator.__del__ at 0x73016398>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 100, in __del__
    self.close()
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 92, in close
    self.cleanup()
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 83, in cleanup
    if not self.mapped_buffers_used[fd] and fd not in self.libcamera_fds:

INFO:picamera2.picamera2:Camera stopped
DEBUG:picamera2.picamera2:Requesting configuration: {'use_case': 'still', 'transform': <libcamera.Transform 'hvflip'>, 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'buffer_count': 1, 'queue': True, 'main': {'format': 'BGR888', 'size': (1920, 1080), 'stride': 5760, 'framesize': 6220800}, 'lores': None, 'raw': {'format': 'SRGGB12_CSI2P', 'size': (4056, 3040), 'stride': 6112, 'framesize': 18580480}, 'controls': {'NoiseReductionMode': <NoiseReductionModeEnum.HighQuality: 2>, 'FrameDurationLimits': (100, 1000000000), 'Saturation': 0.0, 'Sharpness': 12.0, 'Brightness': 0.1, 'AnalogueGain': 6, 'ColourGains': (0, 0), 'AeExposureMode': <AeExposureModeEnum.Normal: 0>, 'AwbEnable': False, 'AeEnable': False, 'ExposureTime': 10000}, 'sensor': {'bit_depth': 12, 'output_size': (4056, 3040)}, 'display': None, 'encode': None}
[11:24:21.998080685] [9006]  INFO Camera camera.cpp:1183 configuring streams: (0) 1920x1080-BGR888 (1) 4056x3040-SRGGB12_CSI2P
[11:24:22.014301048] [8939]  INFO RPI vc4.cpp:608 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 4056x3040-SRGGB12_1X12 - Selected unicam format: 4056x3040-pRCC
INFO:picamera2.picamera2:Configuration successful!
DEBUG:picamera2.picamera2:Final configuration: {'use_case': 'still', 'transform': <libcamera.Transform 'hvflip'>, 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'buffer_count': 1, 'queue': True, 'main': {'format': 'BGR888', 'size': (1920, 1080), 'stride': 5760, 'framesize': 6220800}, 'lores': None, 'raw': {'format': 'SRGGB12_CSI2P', 'size': (4056, 3040), 'stride': 6112, 'framesize': 18580480}, 'controls': {'NoiseReductionMode': <NoiseReductionModeEnum.HighQuality: 2>, 'FrameDurationLimits': (100, 1000000000), 'Saturation': 0.0, 'Sharpness': 12.0, 'Brightness': 0.1, 'AnalogueGain': 6, 'ColourGains': (0, 0), 'AeExposureMode': <AeExposureModeEnum.Normal: 0>, 'AwbEnable': False, 'AeEnable': False, 'ExposureTime': 10000}, 'sensor': {'bit_depth': 12, 'output_size': (4056, 3040)}, 'display': None, 'encode': None}
DEBUG:picamera2.picamera2:Streams: {'main': <libcamera._libcamera.Stream object at 0x653c10a0>, 'lores': None, 'raw': <libcamera._libcamera.Stream object at 0x653c17e0>}
ERROR:root:34
DEBUG:picamera2.picamera2:Requesting configuration: {'use_case': 'preview', 'buffer_count': 4, 'transform': <libcamera.Transform 'identity'>, 'display': 'main', 'encode': 'main', 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'controls': <Controls: {'NoiseReductionMode': <NoiseReductionModeEnum.Minimal: 3>, 'FrameDurationLimits': (100, 83333)}>, 'main': {'size': (640, 480), 'format': 'XBGR8888', 'stride': 2560, 'framesize': 1228800}, 'lores': None, 'raw': {'size': (2028, 1520), 'format': 'SBGGR12_CSI2P', 'stride': 3072, 'framesize': 4669440}, 'queue': True, 'sensor': {'bit_depth': 12, 'output_size': (2028, 1520)}}
INFO:picamera2.picamera2:Camera configuration has been adjusted!
[11:24:22.035280584] [9006]  INFO Camera camera.cpp:1183 configuring streams: (0) 640x480-XBGR8888 (1) 2028x1520-SBGGR12_CSI2P
[11:24:22.036419896] [8939]  INFO RPI vc4.cpp:608 Sensor: /base/soc/i2c0mux/i2c@1/imx477@1a - Selected sensor format: 2028x1520-SBGGR12_1X12 - Selected unicam format: 2028x1520-pBCC
INFO:picamera2.picamera2:Configuration successful!
DEBUG:picamera2.picamera2:Final configuration: {'use_case': 'preview', 'buffer_count': 4, 'transform': <libcamera.Transform 'identity'>, 'display': 'main', 'encode': 'main', 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'controls': <Controls: {'NoiseReductionMode': <NoiseReductionModeEnum.Minimal: 3>, 'FrameDurationLimits': (100, 83333)}>, 'main': {'size': (640, 480), 'format': 'XBGR8888', 'stride': 2560, 'framesize': 1228800}, 'lores': None, 'raw': {'size': (2028, 1520), 'format': 'SBGGR12_CSI2P', 'stride': 3072, 'framesize': 4669440}, 'queue': True, 'sensor': {'bit_depth': 12, 'output_size': (2028, 1520)}}
DEBUG:picamera2.picamera2:Streams: {'main': <libcamera._libcamera.Stream object at 0x653c10a0>, 'lores': None, 'raw': <libcamera._libcamera.Stream object at 0x653c17e0>}
DEBUG:picamera2:Allocated 4 buffers for stream 0 with fds [33, 36, 41, 48]
DEBUG:picamera2:Allocated 4 buffers for stream 1 with fds [51, 55, 75, 78]
INFO:picamera2.picamera2:Camera started
DEBUG:picamera2.picamera2:Execute job: <picamera2.job.Job object at 0x624b4810>
ERROR:picamera2.previews.null_preview:Exception during process_requests()
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/picamera2/previews/null_preview.py", line 85, in handle_request
    picam2.process_requests(self)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1265, in process_requests
    encoder.encode(encoder.name, req)
  File "/usr/lib/python3/dist-packages/picamera2/encoders/encoder.py", line 210, in encode
    self._encode(stream, request)
  File "/usr/lib/python3/dist-packages/picamera2/encoders/v4l2_encoder.py", line 279, in _encode
    fcntl.ioctl(self.vd, VIDIOC_QBUF, buf)
OSError: [Errno 22] Invalid argument
Exception in thread Thread-4 (thread_func):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/picamera2/previews/null_preview.py", line 29, in thread_func
    callback(picam2)
  File "/usr/lib/python3/dist-packages/picamera2/previews/null_preview.py", line 85, in handle_request
    picam2.process_requests(self)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1265, in process_requests
    encoder.encode(encoder.name, req)
  File "/usr/lib/python3/dist-packages/picamera2/encoders/encoder.py", line 210, in encode
    self._encode(stream, request)
  File "/usr/lib/python3/dist-packages/picamera2/encoders/v4l2_encoder.py", line 279, in _encode
    fcntl.ioctl(self.vd, VIDIOC_QBUF, buf)
OSError: [Errno 22] Invalid argument

Hardware : The problem appear with imx477 and not with imx708 on raspberry 2 and 3 with the latest firmware (rpi-update).

Hi, thanks for the report. Reading through the code it looks like you're starting a video recording, and then while the video recording is still running, you're changing the camera mode for a still capture and then switching back to the video mode. Is that correct?

Unfortunately this isn't supported - it just won't survive having all the video buffers being wiped out and reallocated under its feet. I'm a bit surprised it doesn't spit out an error to that effect, so maybe that's something to look at.

Having said that, the next release of Picamera2 (on the "next" branch) will feature a "persistent allocator" whereby you can stop the buffers being deallocated, and force it to reuse the same ones when the video mode restarts. That might actually work better, though there's still an issue about requests be recreated during configure(). So worth a try, but no promises. @will-v-pi What do you think?

Hi, thanks for the report. Reading through the code it looks like you're starting a video recording, and then while the video recording is still running,

Yes - like show in still_during_video.py.

you're changing the camera mode for a still capture and then switching back to the video mode. Is that correct?

Yes - basically I want to take pictures with another (higher) resolution that the video stream.

Unfortunately this isn't supported - it just won't survive having all the video buffers being wiped out and reallocated under its feet. I'm a bit surprised it doesn't spit out an error to that effect, so maybe that's something to look at.

Having said that, the next release of Picamera2 (on the "next" branch) will feature a "persistent allocator" whereby you can stop the buffers being deallocated, and force it to reuse the same ones when the video mode restarts. That might actually work better, though there's still an issue about requests be recreated during configure(). So worth a try, but no promises. @will-v-pi What do you think?

I do not see why: But using my full code which has basically not changed and which should be quite similar to my example code above it works (using next branch). The full code just set some configurations of camera control etc. where restarting the camera several times.

By the way: Isn't picam2.still_configuration automatically used when calling picam2.capture_imageso that I do not have to reconfigure?

Update:

The example code some time crahsed only after 4 to 5 iterations.
My full program is now crashing during picture taking but after several time reconfigurating video

    self._camera.configure(self.preview_configuration)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1112, in configure
    self.configure_(camera_config)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1095, in configure_
    self.allocator.allocate(libcamera_config, camera_config.get("use_case"))
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 32, in allocate
    self.cleanup()
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 86, in cleanup
    del self.mapped_buffers_used[fd]
        ~~~~~~~~~~~~~~~~~~~~~~~~^^^^

My full program is now also crashed at picture taking again after doing several video reconfigures with

request.release()
File "/usr/lib/python3/dist-packages/picamera2/request.py", line 140, in release
self.picam2.allocator.release(self.request.buffers)
File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 78, in release
self.cleanup()
File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 83, in cleanup
if not self.mapped_buffers_used[fd] and fd not in self.libcamera_fds:
       ~~~~~~~~~~~~~~~~~~~~~~~~^^^^
KeyError: 30

I will retest with the next release.

Hi again, there's quite a fundamental thing about cameras in that they run continuously, in one particular readout mode. The cameras themselves are quite pipelined, and so is all the software. You can't just change readout mode without stopping everything, waiting for everything to finish, and then starting over.

You can take still captures while a video is running, but only if you don't change the camera mode, which is what the example program does. You can choose a camera mode with a faster framerate for video, but then still resolution is limited, or you can choose a higher resolution mode, but the video framerate will be limited. There's no getting around this as things stand (until we one day have high framerate high resolution camera modes).

As I said, you may have more luck with the next release if you adjust your code to use the persistent allocator correctly, but I can't promise that because the original design did require you to stop everything when you reconfigure the camera.

In my code example above I already stop the camera before reconfiguring and that is OK for me that the video stream will stop for approximate 1 s.

In my full application it works now with > 12 pictures taken during video. The problems seem s to appear when reconfiguring video via

        li = list(camera.encoders)
        if camera.encoders is not None\
                and len(camera.encoders) > 0\
                and li[0].running:
            camera.stop_recording()
            camera.configure(configuration)
            camera.start_encoder(encoder=encoder)
            camera.start()
        else:
            camera.configure(configuration)

Also picture resolutions other than 1920x1080 or (5074, 3804) but with a ratio of 507 to 380 cause problems - even just run the camera for picture taking they are working.

P.S.: Some background about my application: I am taking and processing pictures which are triggers by network signals. In addition I have a web interface which shows a live stream where one can set camera settings and view the effect directly. The gui is like this - less powerful but also for picamera2 and included e.g. an exposure time setting which takes current sunup/down times into account. Maybe a can publish it in the future. picamera2 really has a lot of features and works very well - at least for ixm708. Thanks a lot for your efford.

The PersistentAllocator on the next branch would be useful for your use case, provided you have enough memory, as it will avoid reallocating the buffers every time you call configure, so you might want to try it out. To try it just add the import from picamera2.allocators import PersistentAllocator and use picam2 = Picamera2(allocator=PersistentAllocator()) to create your picam2 object

Thanks a lot - With my example code it works very well - tested up to more than 100 loop.

With my full software on a Raspberry Pi 2 Model B Rev 1.1 with only 1 GB RAM I got:

FO:picamera2.picamera2:Camera started
DEBUG:picamera2.picamera2:Execute job: <picamera2.job.Job object at 0x652c9b50>
[1:00:51.143660070] [1689] ERROR V4L2 v4l2_videodevice.cpp:1697 /dev/video14[23:cap]: Failed to queue buffer 0: Invalid argument
[1:00:51.144703499] [1689] ERROR RPISTREAM rpi_stream.cpp:276 Failed to queue buffer for ISP Output0
[1:00:51.146265518] [1689] ERROR V4L2 v4l2_videodevice.cpp:1697 /dev/video14[23:cap]: Failed to queue buffer 1: Invalid argument
[1:00:51.147047178] [1689] ERROR RPISTREAM rpi_stream.cpp:276 Failed to queue buffer for ISP Output0
[1:00:51.148504666] [1689] ERROR V4L2 v4l2_videodevice.cpp:1697 /dev/video14[23:cap]: Failed to queue buffer 2: Invalid argument
[1:00:51.149259348] [1689] ERROR RPISTREAM rpi_stream.cpp:276 Failed to queue buffer for ISP Output0
[1:00:51.150765065] [1689] ERROR V4L2 v4l2_videodevice.cpp:1697 /dev/video14[23:cap]: Failed to queue buffer 3: Invalid argument
[1:00:51.151552194] [1689] ERROR RPISTREAM rpi_stream.cpp:276 Failed to queue buffer for ISP Output0
[1:00:51.152979630] [1689] ERROR V4L2 v4l2_videodevice.cpp:1697 /dev/video14[23:cap]: Failed to queue buffer 4: Invalid argument
[1:00:51.153738842] [1689] ERROR RPISTREAM rpi_stream.cpp:276 Failed to queue buffer for ISP Output0
[1:00:51.155146227] [1689] ERROR V4L2 v4l2_videodevice.cpp:1697 /dev/video14[23:cap]: Failed to queue buffer 5: Invalid argument
[1:00:51.155899866] [1689] ERROR RPISTREAM rpi_stream.cpp:276 Failed to queue buffer for ISP Output0
[1:00:51.162895747] [1689] FATAL default v4l2_videodevice.cpp:1960 /dev/video14[23:cap]: assertion "cache_->isEmpty()" failed in streamOff()
Backtrace:
libcamera::V4L2VideoDevice::streamOff()+0x194 (/usr/lib/arm-linux-gnueabihf/libcamera.so.0.1.0 [0x00000000747d7534])
libcamera::RPi::PipelineHandlerBase::stopDevice(libcamera::Camera*)+0x38 (/usr/lib/arm-linux-gnueabihf/libcamera.so.0.1.0 [0x00000000747fce20])
libcamera::PipelineHandler::stop(libcamera::Camera*)+0x38 (/usr/lib/arm-linux-gnueabihf/libcamera.so.0.1.0 [0x00000000747ac0a4])
libcamera::Object::message(libcamera::Message*)+0x8c (/usr/lib/arm-linux-gnueabihf/libcamera-base.so.0.1.0 [0x00000000746d1dc0])
libcamera::Thread::dispatchMessages(libcamera::Message::Type)+0x1cc (/usr/lib/arm-linux-gnueabihf/libcamera-base.so.0.1.0 [0x00000000746d4564])
libcamera::EventDispatcherPoll::processEvents()+0x34 (/usr/lib/arm-linux-gnueabihf/libcamera-base.so.0.1.0 [0x00000000746ca7f0])
libcamera::Thread::exec()+0x7c (/usr/lib/arm-linux-gnueabihf/libcamera-base.so.0.1.0 [0x00000000746d3ee4])
libcamera::CameraManager::Private::run()+0xb8 (/usr/lib/arm-linux-gnueabihf/libcamera.so.0.1.0 [0x000000007477d348])
??? [0x000000007507e068] (/usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.30 [0x000000007507e068])
start_thread+0x164 (./nptl/pthread_create.c:443)

But you already said "provided you have enough memory".

Would it also help to fully close the camera (picam2.close()) and delete the object (delete My_camera_object) each time before recording video is restarted?

Because of #887 I also tested this on the next branch - but with similar errors.

Generally I'd probably try to leave resources allocated if you can. Closing everything runs the risk that you can't reopen the camera, or fail to allocate resources. But it always depends on your application.

Just to be sure: Of course this project is the picamera2 and not the libcameralibrary.

But as I understand all the issues about memory allocation etc. should be caused by libcamera it self. Thus, using the libcamera C++ library directly would not improve anything. (?)

Of course having a compiled C++ program, after existing it all resources should be freed again.

When you run Picamera2 applications, the buffer allocation is under the control of Picamera2. We used to ask libcamera to allocate buffers for us but now we allocate them ourselves and pass them into libcamera. The lifetime of those buffers is under the control of Picamera2 as well.

If you code C++ appliications directly it's exactly the same. You have to decide where to allocate the buffers from, and you can ask libcamera for them (which is what we used to do in our rpicam-apps), though here too we have switched to allocating them ourselves (just as Picamera2 does). Basically the situation is the same in both cases.

raspberrypi / picamera2

take pictures during a video recording -errors #930