Stuck in "with dai.Device(self.pipeline) as device:" when using multiprocessing

[Embeddediot]

Hello I have been trying to implement multiprocessing in order to receive images from a OAK-D camera and a genicam simultaneously. My program is however not able to run the line "with dai.Device(self.pipeline) as device:" that i have from the rgb_preview example. The program runs fine when the oak camera is run on the main thread, and works fine when multithreading is implemented. The real mysterious part is that multiprocessing works fine when used in the debugger. Have you experiences anything similar or been able to use multiprocessing with the depthai sdk? Best Regarads The main module

import threading
import time
import multiprocessing
from jai import Camera_genicam

def main():

    # Instanciate classes
    oak_cam_1 = Camera_oak()
    oak_cam_1.camera_parameters(1920,1080,20)
    genicam = Camera_genicam()
    genicam.camera_parameters(0,0,1920,1080,18,"BayerGR8")
    start_time = time.time()

    p1 = multiprocessing.Process(target=oak_cam_1.send_data)
    p2 = multiprocessing.Process(target=genicam.save_data)

    p1.start()
    p2.start()

    p1.join()
    p2.join()
    print(f'Process p is alive: {p1.is_alive()}')
    #print(f'Process p is alive: {p2.is_alive()}')
    print("--- %s seconds ---" % (time.time() - start_time))

if __name__ == "__main__":
    main()

The oak.py module:

import cv2
import depthai as dai

# Create pipeline

class Camera_oak:
    def __init__(self) -> None:
        self.pipeline = dai.Pipeline()
    def camera_parameters(self, width, height,fps):

        # Define source and output
        camRgb = self.pipeline.create(dai.node.ColorCamera)
        xoutVideo = self.pipeline.create(dai.node.XLinkOut)
        xoutVideo.setStreamName("video")
        #print(camRgb)                                       #only used to avoid error
        # Properties
        camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
        camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
        camRgb.setFps(fps)
        camRgb.setVideoSize(width, height)
        xoutVideo.input.setBlocking(False)
        xoutVideo.input.setQueueSize(1)

        # Linking
        camRgb.video.link(xoutVideo.input)
        print("The Oak camera is initialized")
        print("Resolution: %dx%d" %(width, height))
        print("FPS : %d" %fps)
    def send_data(self):
      #The line where it is stuck when using multiprocessing:
        with dai.Device(self.pipeline) as device:
            video = device.getOutputQueue(name="video", maxSize=1, blocking=False)
            num=0
            while num<50:
                num+=1
                videoIn = video.get()
                #dai.ImgFrame.getFrame()
                #cv2.imwrite('pic_oak'+str(num)+'.jpg',videoIn.getCvFrame())
                print(videoIn)
                # Get BGR frame from NV12 encoded video frame to show with opencv
                # Visualizing the frame on slower hosts might have overhead

            # cv2.imshow("video", videoIn.getCvFrame())

                #if cv2.waitKey(1) == ord('q'):
                #   break

[Luxonis-Brandon]

Will need @Erol444 help here.

[themarpe]

@Embeddediot

First thing that comes to mind is that both processes started by the multiprocessing module, get to try to run on same device, which locks things up. Can you add a delay between starting one and another process for each cameras and run as is?

[Embeddediot]

Hello @themarpe Thank u for your reply but unfortunately this made no changes in the program.

[Embeddediot]

I have also tried only running the process with the oak camera, but with no luck

[Embeddediot]

Additional update: It works everytime i restart the oak camera, but the program is not able to run multiple times without pulling the plug from the oak camera

[themarpe]

@Embeddediot Can you share the logs with DEPTHAI_LEVEL=debug set?

Make sure you are using the latest depthai library, >=2.15 introduced some fixes WRT requiring to power cycle the device

[Embeddediot]

Yes of course, this is the log when i try to run both cameras so the program wont end but just stop after 50 pictures from the genicam. As far as i can see there are no obvious errors in the log I have updated today and am running version 2.15.4.0.

Thank you for your time it is much appreciated.

terminal-output (1).txt

[spgt902]

Hello,

Wondering it anyone has a fix for this yet? I'm running into the same issue right now.

[Erol444]

Hi @spgt902 , are you running multi-cam setup? If that's the case, you would likely need a ~1 sec delay between initializing cameras, otherwise xlink could crash. Thanks, Erik

[spgt902]

Hi @Erol444

I'm only using one camera but I want to run it in a different process separate from another process and only send measured data from the camera to the other process.

Attached is the device before making the connection

but at "with dai.Device(pipeline) as device; --> it gives the error

[Erol444]

Hi @spgt902 , you can only run one pipeline on the device - so you can't connect to it twice. I would suggest adding the measurement data to the original pipeline. Thanks, Erik

[spgt902]

Gotcha, thank you.

[alexovai]

Is there a resolution for this issue ? I cannot run the pipeline in the main thread and it has to run in separate process. The module we have build is working fine for several camera devices such as RTSP cam, webcam, picam etc. However when we try to add oak-cam as a separate process it gets stuck here.

[2023-04-15 07:59:41.078] [debug] Python bindings - version: 2.21.2.0 from build: 2023-04-05 20:29:02 +0000 [2023-04-15 07:59:41.078] [debug] Library information - version: 2.21.2, commit: 125feb8c2e16ee4bf71b7873a7b990f1c5f17b18 from 2023-04-05 21:24:25 +0300, build: 2023-04-05 20:04:37 +0000 [2023-04-15 07:59:41.080] [debug] Initialize - finished [2023-04-15 07:59:41.165] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.24.tar.xz' open: 2ms, archive read: 84ms [2023-04-15 07:59:41.532] [debug] Resources - Archive 'depthai-device-fwp-3575b77f20e796b4e79953bf3d2ba22f0416ee8b.tar.xz' open: 2ms, archive read: 451ms [2023-04-15 07:59:43.406] [debug] Device - OpenVINO version: 2022.1 [2023-04-15 07:59:43.406] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"imu":null,"logDevicePrints":null,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"maxSpeed":4,"pid":63035,"vid":999},"usb3PhyInternalClock":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null} libnop: 0000: b9 11 b9 05 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 b9 02 00 01 ba 00 be be bb 00 bb 00 be be be 0020: be be be be 00 bb 00 be

Unfortunately our code has several other downstream process which are all running in a different process and will be difficult to change at this moment. Happy to provide more information if needed.

[Erol444]

@alexovai how about just connecting to the device in main thread, then using separate process for getting messages (via callbacks/queues) from the device - would that work?

[alexovai]

Yes, that is what we are thinking as a workaround now. Although the design pattern have to change for oak vs other camera we might go with it if that is the only alternative. My concern is that in python thread is still under GIL and runs on the same core as the main thread. We wanted camera to have a separate core hence multiprocessing.

Out of curiosity why is there such limitation ? This is for my understanding purposes.

[Erol444]

@alexovai it's a known issue, I believe it has something to do with XLink limitation, but my understanding is not deep enough to know the actual culprit.

[themarpe]

@alexovai current workaround is using subprocess & spinning up new interpreter.

Reason for limitation is that some shared library things are shared and remain "intiailized" while the new multiprocess spinoff, isn't aware of it, causing it to get stuck there. - more details to be ironed out later

[Maximilian-Karpowicz]

Hi everyone! I found that importing depthai inside of the function that is going to be used to create the xlinks solved the problem for me! (I've only worked with the one oak d camera at a time though)

So in this instance I'd be running Process(target=run_camera)

def run_camera():
    import depthai as dai

    pipeline = dai.Pipeline()

    colour_cam = pipeline.createColorCamera()
    colour_cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
    colour_cam.setBoardSocket(dai.CameraBoardSocket.CAM_A)

    xout_colour = pipeline.createXLinkOut()
    xout_colour.setStreamName('colour')