Issue using aling filter for C2D and D2C transformations

[Xasyo]

I'm having problems when performing Color to Depth and Depth to Color transformations using new align filter added in 1.10.5 release. It looks like it doesn't take into account lens distortion. However, when using CoordinateTransformHelper::transformationDepthFrameToColorCamera, the result of the transformation is correct. The thing is that I would like to be able to transform images captured from both geometries and CoordinateTransformHelper only supports D2C. Is there a way to perform a C2D transformation without getting weird distoritons or is it intended to be added in the future?

This is how I'm using align filter:

std::shared_ptr<ob::Align> align = std::make_shared<ob::Align>(OB_STREAM_DEPTH);
// Wait up to 100ms for a frameset
auto frameset = pipeline->waitForFrames(100);
// Check if a fameset has been grabbed and if it has color and depth frames in it
if (frameset != nullptr && frameset->colorFrame() != nullptr && frameset->depthFrame() != nullptr) {
    std::shared_ptr<ob::Frame> alignedFrame;
    try {
        alignedFrame = align->process(frameset);
    }
    catch (ob::Error& e) {
        // Handle the exception
    }
    auto alignedFrameSet = alignedFrame->as<ob::FrameSet>();
    auto colorFrame = alignedFrameSet->colorFrame();
    auto depthFrame = alignedFrameSet->depthFrame();

    // Check if there are color and depth frames inside the frameset grabbed
    if (colorFrame != nullptr && depthFrame != nullptr) {

        // Save color and depth data into a CameraFrame
        std::shared_ptr<CameraFrame> frame = std::make_shared<CameraFrame>();

        cv::Mat colorMat(colorFrame->height(), colorFrame->width(), CV_8UC4, colorFrame->data(), cv::Mat::AUTO_STEP);
        cv::cvtColor(colorMat, frame->color, cv::COLOR_BGRA2RGBA);
        cv::imwrite("ColorImage", colorMat);

        cv::Mat depthMat(depthFrame->height(), depthFrame->width(), CV_16U, depthFrame->data(), cv::Mat::AUTO_STEP);
        cv::imwrite("DepthImage", depthMat);
    }
}

This is how im using CoordinateTransformHelper:

// Wait up to 100ms for a frameset
auto frameset = pipeline->waitForFrames(100);
// Check if a fameset has been grabbed and if it has color and depth frames in it
if (frameset != nullptr && frameset->colorFrame() != nullptr && frameset->depthFrame() != nullptr) {

    auto colorFrame = frameset->colorFrame();
    auto depthFrame = frameset->depthFrame();

    // Check if there are color and depth frames inside the frameset grabbed
    if (colorFrame != nullptr && depthFrame != nullptr) {

        // Temporary depth to color transformation since align doesn't seem to take into account lens distortion
        std::shared_ptr<ob::VideoFrame> transDepthFrame = (ob::CoordinateTransformHelper::transformationDepthFrameToColorCamera(device, depthFrame, colorFrame->width(), colorFrame->height()))->as<ob::VideoFrame>();

        if (transDepthFrame != nullptr) {
            // Save color and depth data into a CameraFrame
            std::shared_ptr<CameraFrame> frame = std::make_shared<CameraFrame>();

            cv::Mat colorMat(colorFrame->height(), colorFrame->width(), CV_8UC4, colorFrame->data(), cv::Mat::AUTO_STEP);
            cv::cvtColor(colorMat, frame->color, cv::COLOR_BGRA2RGBA);
            cv::imwrite("ColorImage", colorMat);

            cv::Mat depthMat(transDepthFrame->height(), transDepthFrame->width(), CV_16U, transDepthFrame->data(), cv::Mat::AUTO_STEP);
            cv::imwrite("DepthImage", depthMat);

        }
    }
}

[zhonghong322]

I'm having problems when performing Color to Depth and Depth to Color transformations using new align filter added in 1.10.5 release. It looks like it doesn't take into account lens distortion. However, when using CoordinateTransformHelper::transformationDepthFrameToColorCamera, the result of the transformation is correct. The thing is that I would like to be able to transform images captured from both geometries and CoordinateTransformHelper only supports D2C. Is there a way to perform a C2D transformation without getting weird distoritons or is it intended to be added in the future?

This is how I'm using align filter:

std::shared_ptr<ob::Align> align = std::make_shared<ob::Align>(OB_STREAM_DEPTH);
// Wait up to 100ms for a frameset
auto frameset = pipeline->waitForFrames(100);
// Check if a fameset has been grabbed and if it has color and depth frames in it
if (frameset != nullptr && frameset->colorFrame() != nullptr && frameset->depthFrame() != nullptr) {
    std::shared_ptr<ob::Frame> alignedFrame;
    try {
        alignedFrame = align->process(frameset);
    }
    catch (ob::Error& e) {
        // Handle the exception
    }
    auto alignedFrameSet = alignedFrame->as<ob::FrameSet>();
    auto colorFrame = alignedFrameSet->colorFrame();
    auto depthFrame = alignedFrameSet->depthFrame();

    // Check if there are color and depth frames inside the frameset grabbed
    if (colorFrame != nullptr && depthFrame != nullptr) {

        // Save color and depth data into a CameraFrame
        std::shared_ptr<CameraFrame> frame = std::make_shared<CameraFrame>();

        cv::Mat colorMat(colorFrame->height(), colorFrame->width(), CV_8UC4, colorFrame->data(), cv::Mat::AUTO_STEP);
        cv::cvtColor(colorMat, frame->color, cv::COLOR_BGRA2RGBA);
        cv::imwrite("ColorImage", colorMat);

        cv::Mat depthMat(depthFrame->height(), depthFrame->width(), CV_16U, depthFrame->data(), cv::Mat::AUTO_STEP);
        cv::imwrite("DepthImage", depthMat);
    }
}

This is how im using CoordinateTransformHelper:

// Wait up to 100ms for a frameset
auto frameset = pipeline->waitForFrames(100);
// Check if a fameset has been grabbed and if it has color and depth frames in it
if (frameset != nullptr && frameset->colorFrame() != nullptr && frameset->depthFrame() != nullptr) {

    auto colorFrame = frameset->colorFrame();
    auto depthFrame = frameset->depthFrame();

    // Check if there are color and depth frames inside the frameset grabbed
    if (colorFrame != nullptr && depthFrame != nullptr) {

        // Temporary depth to color transformation since align doesn't seem to take into account lens distortion
        std::shared_ptr<ob::VideoFrame> transDepthFrame = (ob::CoordinateTransformHelper::transformationDepthFrameToColorCamera(device, depthFrame, colorFrame->width(), colorFrame->height()))->as<ob::VideoFrame>();

        if (transDepthFrame != nullptr) {
            // Save color and depth data into a CameraFrame
            std::shared_ptr<CameraFrame> frame = std::make_shared<CameraFrame>();

            cv::Mat colorMat(colorFrame->height(), colorFrame->width(), CV_8UC4, colorFrame->data(), cv::Mat::AUTO_STEP);
            cv::cvtColor(colorMat, frame->color, cv::COLOR_BGRA2RGBA);
            cv::imwrite("ColorImage", colorMat);

            cv::Mat depthMat(transDepthFrame->height(), transDepthFrame->width(), CV_16U, transDepthFrame->data(), cv::Mat::AUTO_STEP);
            cv::imwrite("DepthImage", depthMat);

        }
    }
}

What type of device are you using? The Align Filter currently only supports Gemini 330 series devices. It will support all devices in the future.

[zhonghong322]

In future versions, the interface will be unified, and there will be a single interface to implement both D2C and C2D functionality，all Device use the Align Filter.

[Xasyo]

Thanks for the response! I'm using Femtobolt I guess I'll just have to wait ; )