Unable to Update Camera Parameters (White Balance, Exposure) Dynamically

[cesarpgouveia]

Environment: Operating System: Windows 10 DepthAI Version: most up to date till 02/10/2024 OpenCV Version: 4.8.1 Device: OAK-D Compiler: Visual Studio 2022

Summary: I am trying to dynamically update camera parameters such as White Balance and Exposure using key presses while running a pipeline. The parameters only seem to set once, and after that, further adjustments fail to take effect. Sometimes the application crashes.

Here’s a minimal code example to reproduce the issue (code is a little bit messy so if you find it hard to read just tell me and I will send a minimal version of the problem):

// Closer-in minimum depth, disparity range is doubled (from 95 to 190):
static std::atomic<bool> extended_disparity{false};
// Better accuracy for longer distance, fractional disparity 32-levels:
static std::atomic<bool> subpixel{false};
// Better handling for occlusions:
static std::atomic<bool> lr_check{true};

// Custom parameters:
// Color camera sensor resolution
auto colorCameraResolution = dai::ColorCameraProperties::SensorResolution::THE_720_P;
// Color camera color order
auto colorCameraColorOrder = dai::ColorCameraProperties::ColorOrder::BGR;
// Depth camera sensor resolution
auto depthCameraResolution = dai::MonoCameraProperties::SensorResolution::THE_720_P;
// Depth stereo preset mode
auto depthPresetMode = dai::node::StereoDepth::PresetMode::HIGH_ACCURACY;

void SetParameter(std::shared_ptr<dai::Device> device, std::string prop, double value) {

    std::shared_ptr<dai::DataInputQueue> qControl = device->getInputQueue("camControl");

    // Getting the camera control where you can specify the capture parameters
    auto camControl = dai::CameraControl();

    if(prop == "AutoExposure")
    {
        if(value == 1) camControl.setAutoExposureEnable();
        if(value == 0) camControl.setAutoExposureLock(true);
    }
    if(prop == "Gain") 
    {
        std::chrono::microseconds exposureTime = camControl.getExposureTime();
        camControl.setManualExposure(exposureTime, static_cast<int>(value));
    }
    if(prop == "Exposure") {
        int sensitivity = camControl.getSensitivity();
        camControl.setManualExposure(value, sensitivity);
    }
    if(prop == "WhiteBalance") {
        camControl.setManualWhiteBalance(value);
    }
    if(prop == "AutoFocus") {
        camControl.setAutoFocusMode(dai::RawCameraControl::AutoFocusMode::AUTO);
    }
    if(prop == "FocusPosition") {
        camControl.setManualFocus(value);
    }
    if(prop == "Sharpness") {
        camControl.setSharpness(value);
    }

    qControl->send(camControl);
}

int main() {
    // Create pipeline
    dai::Pipeline pipeline;

    // Define sources and outputs

    // Initialize color camera
    auto camRgb = pipeline.create<dai::node::ColorCamera>();
    // Initialize depth camera
    auto monoLeft = pipeline.create<dai::node::MonoCamera>();
    auto monoRight = pipeline.create<dai::node::MonoCamera>();
    auto depth = pipeline.create<dai::node::StereoDepth>();
    // Set two xout for two streams (color and depth)
    auto xoutColor = pipeline.create<dai::node::XLinkOut>();
    auto xoutDepth = pipeline.create<dai::node::XLinkOut>();
    xoutColor->setStreamName("color");
    xoutDepth->setStreamName("depth");

    // Set xin for sending camera configuration parameters (exposure, gain, etc..)
    auto xinControl = pipeline.create<dai::node::XLinkIn>();
    xinControl->setStreamName("camControl");
    xinControl->out.link(camRgb->inputControl);

    // Set color camera properties
    camRgb->setPreviewSize(300, 300);
    camRgb->setBoardSocket(dai::CameraBoardSocket::CAM_A);
    camRgb->setResolution(colorCameraResolution);
    camRgb->setInterleaved(true);
    camRgb->setColorOrder(colorCameraColorOrder);

    // Set depth camera properties
    monoLeft->setResolution(depthCameraResolution);
    monoLeft->setCamera("left");
    monoRight->setResolution(depthCameraResolution);
    monoRight->setCamera("right");

    // Create a node that will produce the depth map (using disparity output as it's easier to visualize depth this way)
    depth->setDefaultProfilePreset(depthPresetMode);
    // Options: MEDIAN_OFF, KERNEL_3x3, KERNEL_5x5, KERNEL_7x7 (default)
    depth->initialConfig.setMedianFilter(dai::MedianFilter::KERNEL_7x7);
    // LR-check is required for depth alignment
    depth->setLeftRightCheck(lr_check);
    depth->setExtendedDisparity(extended_disparity);
    depth->setSubpixel(subpixel);

    // Linking Color
    camRgb->video.link(xoutColor->input);
    // Linking Depth
    monoLeft->out.link(depth->left);
    monoRight->out.link(depth->right);
    depth->disparity.link(xoutDepth->input);

    /*dai::BoardConfig boardConfig;
    boardConfig.watchdogTimeoutMs = 0;

    pipeline.setBoardConfig(boardConfig);*/

    // Connect to device and start pipeline
    auto device = std::make_shared<dai::Device>(pipeline);

    // Output queue color
    auto qColor = device->getOutputQueue("color");
    // Output queue depth will be used to get the disparity frames from the outputs defined above, in this case no queue is used
    auto qDepth = device->getOutputQueue("depth");

    double whiteBalance = 200;
    double exposure = 5000;
    std::string currentParam = "WhiteBalance";  // Default parameter to modify

    while(true) {
        // Get color frame
        std::shared_ptr<dai::ImgFrame> inColor = qColor->get<dai::ImgFrame>();
        std::shared_ptr<dai::ImgFrame> inDepth = qDepth->get<dai::ImgFrame>();
        cv::Mat colorFrame = inColor->getCvFrame();
        cv::Mat depthFrame = inDepth->getFrame();

        // Normalization for better visualization
        depthFrame.convertTo(depthFrame, CV_8UC1, 255 / depth->initialConfig.getMaxDisparity());

        // Display images
        cv::imshow("ColorFrame", colorFrame);
        // Available color maps: https://docs.opencv.org/3.4/d3/d50/group__imgproc__colormap.html
        cv::applyColorMap(depthFrame, depthFrame, cv::COLORMAP_JET);
        cv::imshow("DepthFrame", depthFrame);

        int key = cv::waitKey(1);

        if(key == 'w' || key == 'W') {
            currentParam = "WhiteBalance";
            std::cout << "Selected parameter: WhiteBalance" << std::endl;
        } else if(key == 'e' || key == 'E') {
            currentParam = "Exposure";
            std::cout << "Selected parameter: Exposure" << std::endl;
        }

        if(key == '+' || key == '=') {
            if(currentParam == "WhiteBalance") {
                whiteBalance += 1;
                SetParameter(device, "WhiteBalance", whiteBalance);
                std::cout << "WhiteBalance increased to " << whiteBalance << std::endl;
            } else if(currentParam == "Exposure") {
                exposure += 100;
                SetParameter(device, "Exposure", exposure);
                std::cout << "Exposure increased to " << exposure << std::endl;
            }
        } else if(key == '-') {
            if(currentParam == "WhiteBalance") {
                whiteBalance -= 1;
                SetParameter(device, "WhiteBalance", whiteBalance);
                std::cout << "WhiteBalance decreased to " << whiteBalance << std::endl;
            } else if(currentParam == "Exposure") {
                exposure -= 100;
                SetParameter(device, "Exposure", exposure);
                std::cout << "Exposure decreased to " << exposure << std::endl;
            }
        }

        if(key == 'q' || key == 'Q') {
            break;
        }
    }
    return 0;
}

[jakaskerl]

@cesarpgouveia commands are accepted and applied when I try running your script. Have crashing issues when changing exposure, didn't check why. Why not just use code from https://github.com/luxonis/depthai-core/blob/main/examples/ColorCamera/rgb_camera_control.cpp?

Thanks, Jaka

[cesarpgouveia]

Hey @jakaskerl, thanks for the quick response! I tried using that code, and everything works except for the tryGetAll() function. It seems like it's not available in my DepthAI NuGet package. Was this function only introduced in a more recent version of the DepthAI core?

Here are the versions I'm currently using for DepthAI core and dependencies:

• depthai-core: 2.24.0
• opencv: 4.8.1
• optional: 1.0.0
• libnop: no specific version
• nlohmann: 3.11.3
• CMake: 3.19.3

Note: I also tried pulling the latest version of the depthai-core c++ repo code, rebuilt everything, and created a test project under your solution. Everything worked as expected, so I’m thinking this might be version-related, but you probably know better than I do.

[cesarpgouveia]

@jakaskerl

[cesarpgouveia]

Hey @jakaskerl apparently that code you sent me and the new version (2.28) solved my issue, thanks for the help, closing this issue.