search

luxonis / depthai-core

DepthAI C++ Library
MIT License
235 stars 127 forks source link

[BUG] Camera InputControl clog up when xlink is connected to multiple cameras #699

Open chengguizi opened 1 year ago

chengguizi commented 1 year ago

Describe the bug A clear and concise description of what the bug is.

Minimal Reproducible Example

I will attached the reproducible code in the next reply.

Expected behavior

The key action I would like to take is the following:

controlIn = pipeline.create(dai.node.XLinkIn)
controlIn.setStreamName('control')
controlIn.out.link(monoA.inputControl)
# removing the follow two lines solve the problem
controlIn.out.link(monoB.inputControl)
controlIn.out.link(monoC.inputControl)

Which means, I would like to control all cameras together.

By right, the latency of the control should not be affected by how many inputControl i have tied to the controlIn. But what I observe:

Screenshots

Behaviour when multiple inputControl from different cameras are tied together

frame lacking = 0
Setting manual exposure, time: 8000 iso: 200
frame lacking = 10
Setting manual exposure, time: 1000 iso: 200
frame lacking = 18
Setting manual exposure, time: 8000 iso: 200
frame lacking = 37
Setting manual exposure, time: 1000 iso: 200
frame lacking = 75
Setting manual exposure, time: 8000 iso: 200
frame lacking = 151
Setting manual exposure, time: 1000 iso: 200
frame lacking = 243
Setting manual exposure, time: 8000 iso: 200
frame lacking = 243

Exponential delays showed up

chengguizi commented 1 year ago

The reproducible code is here https://cdn.discordapp.com/attachments/924798001187274752/1068513158009978930/lack_counting_3cam.py

I also make another attempt, spliting all the controls for each camera, but the same problem persist https://cdn.discordapp.com/attachments/924798001187274752/1068513157460537374/lack_counting_3cam_split.py

chengguizi commented 1 year ago

I now makes a reproducible code for OAK-D-LR.

By running the script, and press spacebar to pause the sending of control. We can observe that the script node still receives command after we stop sending.

This indicate somewhere along controlQueueA.send(ctrl) to controlQueueA.send(ctrl), message get queued up. Seems no API can control the behaviour of the queue?

controlInA.setMaxDataSize(1)
controlInA.setNumFrames(4)

didn't work

#!/usr/bin/env python3

import cv2
import depthai as dai

pipeline = dai.Pipeline()

#mono camera
monoA= pipeline.create(dai.node.MonoCamera)
monoA.setFps(30)
#ImageManip node. Capability to crop, resize, warp, … incoming image frames
manipA = pipeline.create(dai.node.ImageManip)
#XLinkOut node. Sends messages over XLink.
manipOutA = pipeline.create(dai.node.XLinkOut)
manipOutA.setStreamName("cama")

#mono camera
monoB= pipeline.create(dai.node.MonoCamera)
monoB.setFps(30)
#ImageManip node. Capability to crop, resize, warp, … incoming image frames
manipB = pipeline.create(dai.node.ImageManip)
#XLinkOut node. Sends messages over XLink.
manipOutB = pipeline.create(dai.node.XLinkOut)
manipOutB.setStreamName("camb")

#mono camera
monoC= pipeline.create(dai.node.MonoCamera)
monoC.setFps(30)
#ImageManip node. Capability to crop, resize, warp, … incoming image frames
manipC = pipeline.create(dai.node.ImageManip)
#XLinkOut node. Sends messages over XLink.
manipOutC = pipeline.create(dai.node.XLinkOut)
manipOutC.setStreamName("camc")

#specify which socket using
monoA.setBoardSocket(dai.CameraBoardSocket.CAM_A) 
monoA.setResolution(dai.MonoCameraProperties.SensorResolution.THE_1200_P)
monoB.setBoardSocket(dai.CameraBoardSocket.CAM_B) 
monoB.setResolution(dai.MonoCameraProperties.SensorResolution.THE_1200_P)
monoC.setBoardSocket(dai.CameraBoardSocket.CAM_C) 
monoC.setResolution(dai.MonoCameraProperties.SensorResolution.THE_1200_P)

manipA.setMaxOutputFrameSize(monoA.getResolutionHeight()*monoA.getResolutionWidth()*3)
manipB.setMaxOutputFrameSize(monoB.getResolutionHeight()*monoB.getResolutionWidth()*3)
manipC.setMaxOutputFrameSize(monoC.getResolutionHeight()*monoC.getResolutionWidth()*3)

script = pipeline.create(dai.node.Script)
script.setScript("""
    import datetime
    while True:
        controlIn = node.io['in'].get()
        node.warn(f"gotten control at ts = {datetime.datetime.now()}")
        node.io['out'].send(controlIn)
""")

#Input for CameraControl message, which can modify camera parameters in runtime
#Linking
#controlXlinkin
controlInA = pipeline.create(dai.node.XLinkIn)
controlInA.setStreamName('controlA')
controlInA.out.link(script.inputs['in'])
script.outputs['out'].link(monoA.inputControl)

script.inputs['in'].setBlocking(False)
script.inputs['in'].setQueueSize(1)

# no effect
controlInA.setMaxDataSize(1)
controlInA.setNumFrames(4)

# no effect
monoA.inputControl.setBlocking(False)

monoA.out.link(manipA.inputImage) 
manipA.out.link(manipOutA.input)
monoB.out.link(manipB.inputImage) 
manipB.out.link(manipOutB.input)
monoC.out.link(manipC.inputImage) 
manipC.out.link(manipOutC.input)

with dai.Device(pipeline) as device:
    qA = device.getOutputQueue(manipOutA.getStreamName(), maxSize=4, blocking=False)
    qB = device.getOutputQueue(manipOutB.getStreamName(), maxSize=4, blocking=False)
    qC = device.getOutputQueue(manipOutC.getStreamName(), maxSize=4, blocking=False)

    # maxSize, blocking plays no effect
    controlQueueA = device.getInputQueue(controlInA.getStreamName(), maxSize=1, blocking=False)

    sensIso=200

    expTime_state1=1000
    expTime_state2=8000
    expTime=1000

    frame_lacking=0

    #ensure exposure_sent == exposure_received
    inA = qA.get()
    exposure_received=int(inA.getExposureTime().total_seconds()*1000000)
    exposure_sent=exposure_received

    paused = False

    while True:

        if not paused:
            inA = qA.get()

            exposure_received=int(inA.getExposureTime().total_seconds()*1000000)

            #received exposure == sent exposure
            if(exposure_received == exposure_sent):   

                print(f"frame lacking = {frame_lacking}")
                #reset
                frame_lacking=0

            else:   #not receiving the sent exposure
                frame_lacking+=1

            #display image
            cv2.imshow("camA", inA.getCvFrame())
        key = cv2.waitKey(1)

        if key == ord(' '):
            paused = not paused
            if paused:
                print("paused")
            else:
                print("continue")

        if paused:
            continue

        if(expTime==expTime_state1 and (exposure_received==exposure_sent)): #at stage 1 and received the sent exposure

            #send command to change to another state
            expTime=expTime_state2
            exposure_sent=expTime
            ctrl = dai.CameraControl()
            ctrl.setManualExposure(expTime, sensIso)
            print("Setting manual exposure, time:", expTime, "iso:", sensIso)
            controlQueueA.send(ctrl)

        elif(expTime==expTime_state2 and (exposure_received==exposure_sent)): #at stage 2 and received the sent exposure

            #send command to change to another state
            expTime=expTime_state1
            exposure_sent=expTime
            ctrl = dai.CameraControl()
            ctrl.setManualExposure(expTime, sensIso)
            print("Setting manual exposure, time:", expTime, "iso:", sensIso)
            controlQueueA.send(ctrl)
        else:
            controlQueueA.send(ctrl)
saulthu commented 1 year ago

In the past I've had a lot of trouble with sending control commands to multiple cameras. Setting controls before the camera has started streaming, and even within the first few frames, can be a problem. I wait to send commands until after 10 frames have arrived, per camera. Probably unnecessary, but I queue all commands in a thread-safe queue, and process all pending commands in a grabbing thread.