Picamera2 failed to turn on while using python multiprocessing

[hillyuyichu]

Please only include one item/question/problem per issue! This is the follow up for my topic on Raspberry pi forum " Picamera2 initialized successfully but failed to configure".

This morning, my picamera2 was able to initialize successfully and configure successfully but failed to turn on (could be problems with the rest of my code). The program did not throw any error. It just stopped. However, the second time I tried to run it. It give me the same error I experienced as mentioned in my pi forum topic, which are " failure to acquire camera" and " initsequence did not complete"

Two errors generated in the console, attached here. error.txt

Using Raspberry pi 4 8GB Arducam 5MP Camera 1080P HD OV5647 Camera Module V1 PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)" NAME="Raspbian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

pi is connected to a screen. But I am operating it headless via VNC viewer.

I think it could be the multiprocessing module because I changed my camera.py, see below.

Current camera.py `import multiprocessing as mp from multiprocessing import JoinableQueue, Event from queue import Empty from time import sleep from picamera2 import Picamera2

import logging

logger = logging.getLogger(name)

class Camera(object): """Control the camera using the Picamera module (threaded)"""

def __init__(self):

    self.file_queue = JoinableQueue(1)

    self.quit_event = Event()

def _worker(self):

    logger.info("Initialising camera...")

    camera = Picamera2()
    camera_config = camera.create_still_configuration(main={"size": (240, 240)})
    camera.configure(camera_config)

    while True:
        try:  # Timeout raises queue.Empty
            file_path = self.file_queue.get(block=True, timeout=0.1)

            logger.debug("Capturing image")
            camera.capture_file(file_path)

            self.file_queue.task_done()

        except Empty:
            if self.quit_event.is_set():
                logger.debug("Quitting camera thread...")
                break

def start(self, file_path):
    logger.debug("Calling start")
    self.file_queue.put(file_path, block=True)

def join(self):
    logger.debug("Calling join")
    self.file_queue.join()

def launch(self):
    logger.debug("Initialising worker")
    self.p = mp.Process(target=self._worker, daemon=True)
    self.p.start()

def quit(self):
    self.quit_event.set()
    self.p.join()

Old camera.py from time import sleep from picamera2 import Picamera2

class Camera(object):

def __init__(self):

    camera = Picamera2()
    camera_config = camera.create_still_configuration(main={"size": (240, 240)})
    camera.configure(camera_config)

    self.camera=camera
    self.camera.start()
    sleep(2)

def capture(self, file_path):

    self.camera.capture_file(file_path)

    return file_path

`

[davidplowman]

Hi, thanks for following this up.

Firstly, I was wondering if I might ask you to fix the formatting so that all the Python code is in the correct style. That would make it a bit easier to read.

Also, I'm still not clear what I need to do to "run" your code. I can import your camera.py, but what do I do then?

Finally, when you talk about "running it for the second time" what does this mean? Are you quitting the Python interpreter and re-starting it? Or are you starting a second Python interpreter? Or making a second Camera object, or calling launch() twice? If you could explain what I need to do to reproduce the errors, that would really help. Thank you very much!

[hillyuyichu]

Sorry, I forgot to mention how I ran this. I run this through an API.py which is my flask to interact with frontend. OnionBot object is called from a main.py that contains camera.py.

API.py:

import os
import sys

from flask import Flask
from flask import request
from flask_cors import CORS

from main import OnionBot

import logging

# Silence Flask werkzeug logger
logger = logging.getLogger("werkzeug")
logger.setLevel(logging.ERROR)  # Note! Set again below

# Initialise OnionBot
bot = OnionBot()
bot.run()

# Initialise flask server
app = Flask(__name__)
CORS(app)

@app.route("/", methods=["GET", "POST"])
def index():
    """Access the OnionBot portal over the local network"""

    if request.form["action"] == "start":
        logger.debug("start called")
        bot.start(request.form["value"])
        return "1"

    if request.form["action"] == "stop":
        logger.debug("stop called")
        return bot.stop()

    if request.form["action"] == "get_latest_meta":
        logger.debug("get_latest_meta called")
        return bot.get_latest_meta()

    if request.form["action"] == "get_thermal_history":
        logger.debug("get_thermal_history called")
        return bot.get_thermal_history()

    if request.form["action"] == "set_label":
        logger.debug("set_label called")
        bot.set_label(request.form["value"])
        return "1"

    if request.form["action"] == "set_no_label":
        logger.debug("set_no_label called")
        bot.set_no_label()
        return "1"

    if request.form["action"] == "set_classifiers":
        logger.debug("set_classifiers called")
        bot.set_classifiers(request.form["value"])
        return "1"

    if request.form["action"] == "get_temperature_setpoint":
        logger.debug("get_temperature_setpoint called")
        return bot.get_temperature_setpoint()

    if request.form["action"] == "get_camera_frame_rate":
        logger.debug("get_camera_frame_rate called")
        return bot.get_camera_frame_rate()

    if request.form["action"] == "set_fixed_setpoint":
        logger.debug("set_fixed_setpoint called")
        bot.set_fixed_setpoint(request.form["value"])
        return "1"

    if request.form["action"] == "set_temperature_target":
        logger.debug("set_temperature_target called")
        bot.set_temperature_target(request.form["value"])
        return "1"

    if request.form["action"] == "set_temperature_hold":
        logger.debug("set_temperature_hold called")
        bot.set_temperature_hold()
        return "1"

    if request.form["action"] == "pi-restart":
        os.system("sudo reboot")

    if request.form["action"] == "pi-shutdown":
        os.system("sudo shutdown now")

    if request.form["action"] == "restart":
        os.system(". ~/onionbot/runonion")

    if request.form["action"] == "quit":
        bot.quit()
        logger.info("Shutting down server")
        server_quit = request.environ.get("werkzeug.server.shutdown")
        if server_quit is None:
            raise RuntimeError("Not running with the Werkzeug Server")
        server_quit()
        sys.exit()
        os.system("sleep 1 ; pkill -f API.py")  # If all else fails...

if __name__ == "__main__":
    app.run(debug=False, host="0.0.0.0")

Main.py:

from threading import Thread, Event
from time import sleep

from thermal_camera import ThermalCamera
from camera import Camera
from cloud import Cloud
from classification import Classify
from data import Data
from config import Settings, Labels

from datetime import datetime
from json import dumps
import logging

# Fix logging faliure issue
for handler in logging.root.handlers[:]:
    logging.root.removeHandler(handler)

# Initialise custom logging format
FORMAT = "%(relativeCreated)6d %(levelname)-8s %(name)s %(process)d %(message)s"
logging.basicConfig(format=FORMAT, level=logging.INFO)
logger = logging.getLogger(__name__)

settings = Settings()
labels = Labels()
camera = Camera()
thermal = ThermalCamera()
cloud = Cloud()
classify = Classify()
data = Data()

class OnionBot(object):
    """Main OnionBot script"""

    def __init__(self):

        self.quit_event = Event()

        # Launch multiprocessing threads
        logger.info("Launching worker threads...")
        camera.launch()
        thermal.launch()
        cloud.launch_camera()
        cloud.launch_thermal()
        classify.launch()

        self.latest_meta = " "
        self.session_ID = None
        self.label = None

        logger.info("OnionBot is ready")

    def run(self):
        """Start logging"""

        def _worker():

            measurement_ID = 0
            file_data = None
            meta = None

            while True:

                # Get time stamp
                timer = datetime.now()

                # Get update on key information
                measurement_ID += 1
                label = self.label
                session_ID = self.session_ID

                # Generate file_data for logs
                queued_file_data = data.generate_file_data(
                    session_ID, timer, measurement_ID, label
                )

                # Generate metadata for frontend
                queued_meta = data.generate_meta(
                    session_ID=session_ID,
                    timer=timer,
                    measurement_ID=measurement_ID,
                    label=label,
                    file_data=queued_file_data,
                    thermal_data=thermal.data,
                    classification_data=classify.database,
                )

                # Start sensor capture
                camera.start(queued_file_data["camera_file"])
                thermal.start(queued_file_data["thermal_file"])

                # While taking a picture, process previous data in meantime
                if file_data:

                    cloud.start_camera(file_data["camera_file"])
                    cloud.start_thermal(file_data["thermal_file"])
                    classify.start(file_data["camera_file"])

                    # Wait for all meantime processes to finish
                    cloud.join_camera()
                    cloud.join_thermal()
                    classify.join()

                    # Push meta information to file level for API access
                    self.labels_csv_filepath = file_data["label_file"]
                    self.latest_meta = dumps(meta)

                # Wait for queued image captures to finish, refresh control data
                thermal.join()
                camera.join()

                # Log to console
                if meta is not None:
                    attributes = meta["attributes"]
                    logger.info(
                        "Logged %s | session_ID %s | Label %s | Interval %0.2f | Temperature %s | PID enabled: %s | PID components: %0.1f, %0.1f, %0.1f "
                        % (
                            attributes["measurement_ID"],
                            attributes["session_ID"],
                            attributes["label"],
                            attributes["interval"],
                            attributes["temperature"],
                        )
                    )

                # Move queue forward one place
                file_data = queued_file_data
                meta = queued_meta

                # Add delay until ready for next loop
                frame_interval = float(settings.get_setting("frame_interval"))
                while True:
                    if (datetime.now() - timer).total_seconds() > frame_interval:
                        break
                    elif self.quit_event.is_set():
                        break
                    sleep(0.01)

                # Check quit flag
                if self.quit_event.is_set():
                    logger.debug("Quitting main thread...")
                    break

        # Start thread
        logger.info("Starting main script")
        self.thread = Thread(target=_worker, daemon=True)
        self.thread.start()

    def start(self, session_ID):
        data.start_session(session_ID)
        self.session_ID = session_ID

    def stop(self):
        """Stop logging"""
        self.session_ID = None
        labels = self.labels_csv_filepath
        cloud.start_camera(
            labels
        )  # Use cloud uploader camera thread to upload label file
        cloud.join_camera()
        return cloud.get_public_path(labels)

    def get_latest_meta(self):                                    #
        """Returns cloud filepath of latest meta.json - includes path location of images"""
        return self.latest_meta

    def get_thermal_history(self):
        """Returns last 300 temperature readings"""
        return self.thermal_history

    def set_label(self, string):
        """Command to change current label -  for building training datasets"""
        self.label = string

    def set_no_label(self):
        """Command to set label to None type"""
        self.label = None

    def set_classifiers(self, string):                        #
        """Command to change current classifier for predictions"""
        classify.set_classifiers(string)

    def set_fixed_setpoint(self, value):                       #
        """Command to change fixed setpoint"""
        control.update_fixed_setpoint(value)

    def set_temperature_target(self, value):                  #
        """Command to change temperature target"""
        control.update_temperature_target(value)

    def set_temperature_hold(self):
        """Command to hold current temperature"""
        control.hold_temperature()

    def quit(self):
        logger.info("Raising exit flag")
        self.quit_event.set()
        self.thread.join()
        logger.info("Main module quit")
        camera.quit()
        logger.info("Camera module quit")
        thermal.quit()
        logger.info("Thermal module quit")
        cloud.quit()
        logger.info("Cloud module quit")
        classify.quit()
        logger.info("Classifier module quit")
        logger.info("Quit process complete")

[hillyuyichu]

The first time I run API.py after having restarted the pi. It produces the following error :

%Run API.py
  2903 INFO     thermal_camera 2109 Initialising thermal camera...
  2968 INFO     cloud 2109 Initialising cloud upload...
  2969 INFO     classification 2109 Initialising classifier...
  2969 INFO     data 2109 Initialising data management...
  2970 INFO     control 2109 Initialising control script...
  2971 INFO     main 2109 Launching worker threads...
  2990 INFO     main 2109 OnionBot is ready
  2990 INFO     camera 2112 Initialising camera...
  2992 INFO     main 2109 Starting main script
[0:06:29.254244875] [2112]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
 * Serving Flask app "API" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
[0:06:29.326740939] [2119]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media4 and ISP device /dev/media1
  3083 INFO     picamera2 2112 Initialization successful.
  3084 INFO     picamera2 2112 Camera now open.
  3084 DEBUG    picamera2 2112 <libcamera._libcamera.CameraManager object at 0xb1702ce0>
  3088 DEBUG    picamera2 2112 Requesting configuration: {'use_case': 'still', 'transform': <libcamera.Transform 'identity'>, 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'buffer_count': 1, 'main': {'format': 'BGR888', 'size': (240, 240), 'stride': 768, 'framesize': 184320}, 'lores': None, 'raw': None, 'controls': {'NoiseReductionMode': <NoiseReductionModeEnum.HighQuality: 2>, 'FrameDurationLimits': (33333, 120000)}, 'display': None, 'encode': None}
[0:06:29.337183212] [2112]  INFO Camera camera.cpp:1035 configuring streams: (0) 240x240-BGR888
[0:06:29.346779384] [2119]  INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
  3104 INFO     picamera2 2112 Configuration successful!
  3104 DEBUG    picamera2 2112 Final configuration: {'use_case': 'still', 'transform': <libcamera.Transform 'identity'>, 'colour_space': <libcamera.ColorSpace 'sYCC'>, 'buffer_count': 1, 'main': {'format': 'BGR888', 'size': (240, 240), 'stride': 768, 'framesize': 184320}, 'lores': None, 'raw': None, 'controls': {'NoiseReductionMode': <NoiseReductionModeEnum.HighQuality: 2>, 'FrameDurationLimits': (33333, 120000)}, 'display': None, 'encode': None}
  3105 DEBUG    picamera2 2112 Streams: {'main': <libcamera._libcamera.Stream object at 0xb1705240>, 'lores': None, 'raw': None}
  3107 DEBUG    picamera2 2112 Allocated 1 buffers for stream 0.

The second time I run API.py. The error:

%Run API.py
  1586 INFO     thermal_camera 3678 Initialising thermal camera...
  1646 INFO     cloud 3678 Initialising cloud upload...
  1647 INFO     classification 3678 Initialising classifier...
  1647 INFO     data 3678 Initialising data management...
  1648 INFO     control 3678 Initialising control script...
  1649 INFO     main 3678 Launching worker threads...
  1664 INFO     camera 3681 Initialising camera...
[0:17:26.285954773] [3681]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
  1683 INFO     main 3678 OnionBot is ready
  1685 INFO     main 3678 Starting main script
  [0:17:26.325606295] [3685]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media4 and ISP device /dev/media1
 1860 INFO     picamera2 2814 Initialization successful.
[0:06:56.905128914] [2814]  INFO Camera camera.cpp:848 Pipeline handler in use by another process
2022-09-27T17:36:05.145Z | ERROR    | Camera __init__ sequence did not complete.
  1861 ERROR    picamera2 2814 Camera __init__ sequence did not complete.
   WARNING: This is a development server. Do not use it in a production deployment.

[davidplowman]

Hi, thanks for sending this extra information. However, I'm wondering if it's possible for you to create a simple test case that shows the problem. There is a lot of code here, involving various modules that I know nothing at all about, and working through everything and figuring it all out could be time consuming for someone not familiar with it.

Maybe you could start by logging all the calls to Picamera2 and then creating the simplest script possible that shows the problem?

Also, going back to the comments I made on the original forum post, I'm quite nervous about the use of the multiprocessing module with the camera system. Remember that it is one process per camera. If you try and start other processes talking to the same camera then errors like Pipeline handler in use by another process are exactly what I would expect. These kinds of errors cannot be "fixed", the application must be designed in such a way that it simply doesn't work like this.

[hillyuyichu]

Thank you for the reply, David!

I think I have found the problem. Since, I was not able to successfully run the program. The bot.quit() button, which contain terminating multiprocess function, did not show up on my webpage. I think I have to call the camera.quit() from camera.py in order to release the camera. Where do you think it's appropriate for me to add the camera.quit () function in the event of an unsuccessful run?

For the "pipline handler in use by another process" error, I don't get them when I have restarted the pi.

start.py:

import os
import sys
from main import OnionBot

bot = OnionBot()
bot.run()
bot.start()
bot.stop()
bot.quit()

main.py:

from threading import Thread, Event
from time import sleep
from camera import Camera
from control import Control
from data import Data

from datetime import datetime
from json import dumps

camera = Camera()
data = Data()

class OnionBot(object):

    def __init__(self):

        self.quit_event = Event()

        # Launch multiprocessing threads

        camera.launch()

        self.latest_meta = " "
        self.session_ID = None
        self.label = None

    def run(self):

        def _worker():

            measurement_ID = 0
            file_data = None
            meta = None

            while True:

                # Get time stamp
                timer = datetime.now()

                # Get update on key information
                measurement_ID += 1
                label = self.label
                session_ID = self.session_ID

                # Generate file_data for logs
                queued_file_data = data.generate_file_data(
                    session_ID, timer, measurement_ID, label
                )

                # Start sensor capture
                camera.start(queued_file_data["camera_file"])

                # While taking a picture, process previous data in meantime
                if file_data:

                    # Push meta information to file level for API access
                    self.labels_csv_filepath = file_data["label_file"]
                    self.latest_meta = dumps(meta)

                # Wait for queued image captures to finish, refresh control data
                camera.join()

                # Check quit flag
                if self.quit_event.is_set():
                    break

        # Start thread
        self.thread = Thread(target=_worker, daemon=True)
        self.thread.start()

    def start(self, session_ID):
        data.start_session(session_ID)
        self.session_ID = session_ID

    def stop(self):
        self.session_ID = None
        labels = self.labels_csv_filepath
        cloud.start_camera(
            labels
        ) 
        cloud.join_camera()
        return cloud.get_public_path(labels)

    def quit(self):
        self.quit_event.set()
        self.thread.join()
        camera.quit()

Camera.py:

import multiprocessing as mp
from multiprocessing import JoinableQueue, Event
from queue import Empty
from time import sleep
from picamera2 import Picamera2

class Camera(object):
    """Control the camera using the Picamera module (threaded)"""

    def __init__(self):

        self.file_queue = JoinableQueue(1)

        self.quit_event = Event()

    def _worker(self):

        camera = Picamera2()
        camera_config = camera.create_still_configuration(main={"size": (240, 240)})
        camera.configure(camera_config)

        while True:
            try: 
                file_path = self.file_queue.get(block=True, timeout=0.1)

                camera.capture_file(file_path)

                self.file_queue.task_done()

            except Empty:
                if self.quit_event.is_set():
                    break

    def start(self, file_path):
        self.file_queue.put(file_path, block=True)

    def join(self):
        self.file_queue.join()

    def launch(self):
        self.p = mp.Process(target=self._worker, daemon=True)
        self.p.start()

    def quit(self):
        self.quit_event.set()
        self.p.join()

[davidplowman]

Hi again, as I said previously, I'm quite nervous about multiple processes trying to talk to the same camera.

If you have to do this, then you must ensure that only one process opens, uses and closes the camera at a time. It's also very important that, whatever happens (whether the calls succeed or fail), the process must release the camera before any other process can start and successfully open it again.

I don't think I can advise you as to where the best place would be in your application as I don't know enough about it, but so long as your camera process terminates then the camera should be released and another process would then successfully be able to use it.

[hillyuyichu]

Ok, thanks! I'm only using one camera for this program and only this file (camera.py) involves multiprocessing. I think the very first time I ran the whole program it threw an error before it reached the multiprocessing.join() line. Would the camera get released after I shut down the program and reboot the Pi?

[davidplowman]

The camera will certainly get released when you reboot your Pi. In fact, the camera will also be released when the process that has been using the camera terminates. I guess it might possible to end up with "zombie" processes that have got stuck holding the camera, in which case you would either have to kill that process or reboot if that doesn't work.

[hillyuyichu]

OK got it. Thank you for taking the time to answer my questions. Really appreciated!

[davidplowman]

Hi again, is there anything else to investigate here? Otherwise I'll consider closing it in a few days, and you are of course always welcome to open a new issue if anything else comes up. Thanks!

[meawoppl]

As a heads up you have the potential for both flask threads and multiprocess access of the camera in this use case. Some substantial fraction of the picamera2.Camera calls are not safe for concurrency right now, and with the extra flask layer are very very likely to have errors/tracebacks eaten due to some, "sophisticated" choices that flask has made about logging: https://github.com/pallets/flask/issues/2998

I am actively working to make picamera2 smarter about concurrent calls, but in the meantime, I would strongly recommend that you use a multiprocessing.Lock() instance (threading.Lock() will NOT do here) to protect calls into the camera.

Having fiddled a bunch with this library, libcamera, and Python's multiprocessing, I would attach a strong CAVE CANEM note to this use-case, but I am impressed that it is working well!

[hillyuyichu]

I'm mostly good. My multiprocessing is working now as long as I release it everyday I'm done using it. The errors would still occur sometimes if a glitch happened before I even have the chance to turn it off. But in those cases, I just reboot the pi again.

[hillyuyichu]

As a heads up you have the potential for both flask threads and multiprocess access of the camera in this use case. Some substantial fraction of the picamera2.Camera calls are not safe for concurrency right now, and with the extra flask layer are very very likely to have errors/tracebacks eaten due to some, "sophisticated" choices that flask has made about logging: pallets/flask#2998

I am actively working to make picamera2 smarter about concurrent calls, but in the meantime, I would strongly recommend that you use a multiprocessing.Lock() instance (threading.Lock() will NOT do here) to protect calls into the camera.

Having fiddled a bunch with this library, libcamera, and Python's multiprocessing, I would attach a strong CAVE CANEM note to this use-case, but I am impressed that it is working well!

Thanks for the advice! I tried this but it is throwing me a traceback. " lock is not an iterable"

[hillyuyichu]

I think the biggest problem with multiprocessing using Picamera2 now is how we can make sure the camera gets released if an error occurs before the .release() line. This would make the camera much easier to work with.

[davidplowman]

Hi, thanks for the update. I am intending to push another release shortly which does improve some error handling already, but obviously if you can report specific errors that you encounter and which cause problems then I will of course look into those first. Thanks!

[kodonnell]

Here's a simple minimal test. Keep this running in a REPL:

from picamera2 import Picamera2
import time

picam2 = Picamera2()
camera_config = picam2.create_still_configuration(main={"size": (320, 240)})
picam2.configure(camera_config)
picam2.start()
time.sleep(2)
picam2.capture_file("/tmp/tmp.png")
picam2.close()
del picam2

time.sleep(1000)

And once it's sleeping, go run something else that wants the camera e.g. libcamera-still /tmp/out.jpg. In my case, it'll fail, but then work as soon as I close the original python REPL. This seems to indicate there's something not being cleanly released in Python.

NB:

• Version latest with apt-get install python3-picamera2 i.e. 0.3.3-1.
• I can reopen the camera again in python multiple times (e.g. just run that REPL twice).

It's possible this is worth renaming to "Camera not being correctly closed/released" or similar. Note #130 may be similar.

[davidplowman]

@kodonnell Thanks for reporting this problem with a nice simple script to reproduce it!

We have been aware of problems in this area and have made some fixes in the most recent version. Could you try this with the latest version (0.3.5). I've done this and it seems to work for me with the new version. Thanks!

[kodonnell]

I just rebuilt my system image last night, and that pulled in 0.3.5 - and problem gone = ) Good work everyone.

[davidplowman]

@kodonnell Thanks for the confirmation!

I think I'm going to close this issue now as the thread has got fairly long I'm not quite sure what else there is to look. But if folks do have issues in this area then please continue to file new issues for them. As always, when there's a simple script or instructions to reproduce a problem then that really helps us to resolve problems. Thanks to everyone for contributing on this one.