[HOW-TO] use mediapipe with picamera2

Problem between libcamera and mediapipe:

Objective: To be able to use mediapipe on a live video stream from my fisheye camera

Hardware: Raspberry Pi4, Arducam Camera (B0286 Fisheye Lens)

Constraint: The camera only seems to work with the libcamera and picamera2 libraries (which are linked)

Environment: Miniforge3 with Python 3.9

Library: OpenCV, libcamera, picamera2 and mediapipe

Installation : https://docs.arducam.com/Raspberry-Pi-Camera/Native-camera/Libcamera-User-Guide/

Problem: Using mediapipe on pre-recorded video works, using my camera works (fetches frames with picamera2 library and I can even process them with OpenCV behind). But together here is the error :

Codes: To see the problem more simply I separated my code into two parts, one part recovering the frames of my live camera and the other which uses mediapipe in order to detect zero, one or two hands.

Processing of a frame by mediapipe (tested):

Retrieves frames and calls the process_image function in the previous code: To use the picamera2 library, I need the #!/usr/bin/python3 directive but with this directive the mediapipe library no longer works.

Hi @Theo-sierra,

To use the picamera2 library, I need the #!/usr/bin/python3 directive but with this directive the mediapipe library no longer works.

The script is literally requiring to use a specific interpreter and not the one of your miniconda environment. Remove the shebang (#!/usr/bin/python3) and install picamera2 in your Vision environment (see README).

Or install OpenCV and Mediapipe “system-wide”.

P.S.: a more general comment, but sharing code screenshots is not really useful. First, I'm not going to rewrite your code to test it, second, some people are visually impaired, or blind and cannot read your code if it's a screenshot. What you should do is use the code blocks:

```python
# Your amazing code

def easy_to_copy():
   pass


Which will be rendered like so:

```python
# Your amazing code

def easy_to_copy():
   pass

Hi @Vincent-Stragier, First, thank you for your answear and advices about how I have to share my issues ! The problem therefore comes well from what I thought (the shebang) but when I test the code without this line "#! usr/bin/python3", the error comes that the picamera2 library does not find libcamera:

(vision) nicolas@raspberrypi:~/Documents/Vision/Mediapipe $ python3 mediapipe_cam_lib2.py 
Traceback (most recent call last):
  File "/home/nicolas/Documents/Vision/Mediapipe/mediapipe_cam_lib2.py", line 2, in <module>
    from picamera2 import Picamera2
  File "/home/nicolas/miniforge3/envs/vision/lib/python3.9/site-packages/picamera2/__init__.py", line 3, in <module>
    import libcamera
ModuleNotFoundError: No module named 'libcamera'

Besides, this is how I installed libcamera (on environment base and vision) :

with arducam : https://docs.arducam.com/Raspberry-Pi-Camera/Pivariety-Camera/Libcamera-User-Guide/

And I also tried this two solutions without success :

with the libcamera site (Get Started) : https://libcamera.org/getting-started.html
with the Python documentation : https://www.raspberrypi.com/documentation/computers/camera_software.html#building-libcamera-and-libcamera-apps

I give the previous codes in an easy way to copy for you as indicated:

Processing of a frame by mediapipe (tested):

import cv2
import mediapipe as mp

def preprocess_image(image):
    # Réduire la taille de l'image à des dimensions appropriées
    resized_image = cv2.resize(image, (224, 224))
    return resized_image

def process_image(image):

    mp_drawing = mp.solutions.drawing_utils
    mp_hands = mp.solutions.hands

    image = preprocess_image(image)
    height, width, _ = image.shape
    image = cv2.flip(image, 1)
    image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    with mp_hands.Hands(
        static_image_mode=False,
        max_num_hands=2,
        min_detection_confidence=0.5) as hands:

        results = hands.process(image_rgb)

        if results.multi_hand_landmarks is not None:
            # Dessiner les points et les connexions à l'aide de mp_drawing
            for hand_landmarks in results.multi_hand_landmarks:
                mp_drawing.draw_landmarks(
                    image, hand_landmarks, mp_hands.HAND_CONNECTIONS,
                    mp_drawing.DrawingSpec(color=(255, 255, 0), thickness=4, circle_radius=5),
                    mp_drawing.DrawingSpec(color=(255, 0, 255), thickness=4))
        cv2.imshow("Image", image)
        cv2.waitKey(0)

    return image

# Test
im = cv2.imread('image.jpg')
image = process_image(im)

Retrieves frames and calls the process_image function in the previous code:


import cv2
from picamera2 import Picamera2
from mediapipe_cam_med2 import process_image

cv2.startWindowThread()
size = (3280, 2464)

# Initialisation de la caméra
picam2 = Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"size": size}, buffer_count=2))
picam2.start()

# Boucle principale
while True:
    # Capture de l'image en direct de la caméra
    image = picam2.capture_array()

    # Traitement de l'image en utilisant les fonctions de mediapipe
    processed_image = process_image(image)

    # Affichage de l'image traitée
    cv2.imshow("Processed Image", processed_image)
    #cv2.imshow("Processed Image", image)

    # Vérification de la touche 'q' pour quitter la boucle
    if cv2.waitKey(1) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

# Arrêt de la caméra et libération des ressources
cv2.destroyAllWindows()
picam2.stop()

For the python picamera2 package, it is best to use the python3-picamera2 package within Pi OS Bookworm. Using pip install may encounter many issues (because picamera2 depends on libcamera). The picamera2 manual mentions: For users needing to do so, Picamera2 can also be installed using pip. However, we do not recommend this because the underlying libcamera library is not ABI (Application Binary Interface) stable, so you can end up with versions of Picamera2 and libcamera that do not work together. If you always install using apt then all these dependencies are managed for you.

The virtual environment I created using python -m venv --system-site-packages <env-name>

picam2 = Picamera2()
picam2.configure(picam2.create_preview_configuration(main={"format": 'RGB888', "size": (640, 480)}, transform=Transform(hflip=1)))
picam2.start()

while True:
    image = picam2.capture_array()
    mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)

In a conda environment, you might be able to use export PYTHONPATH=$PYTHONPATH:/path/to/system/site-packages to include the system site-packages.

raspberrypi / picamera2

[HOW-TO] use mediapipe with picamera2 #755