save RGBD video data to disk in 30 FPS

[bowieshi]

Hi. I am trying to save Orbbec femto bolt output color datastream and depth datastream to my disk and met some problems. I mainly used the code from example examples/cpp/Sample-SaveToDisk and examples/cpp/Sample-SyncAlignViewer. I enable the software depth to color alignment. I first use the OB_PROFILE_DEFAULT for the color and depth stream and used the saveColor and saveDepth from examples/cpp/Sample-SaveToDisk. I can get the color and depth data in about 15 FPS. I used the window app to show the color and depth overlap viewer and it renders very smooth. Then, I change the setting to use 30 FPS color and depth datastream, like following: colorProfile = colorProfiles->getVideoStreamProfile(3840, 2160, OB_FORMAT_RGB, 30); depthProfile = depthProfiles->getVideoStreamProfile(640, 576, OB_FORMAT_Y16, 30); Then, I run the following code to read and save color and depth data ` while(true) { if (!keyEventProcess(app, pipeline, config)) { break; } // Wait for up to 100ms for a frameset in blocking mode. auto frameset = pipeline.waitForFrames(1000); if(frameset == nullptr) { std::cout << "The frameset is null!" << std::endl; continue; }

    // Filter the first 5 frames of data, and save it after the data is stable
    frameCount++;
    if(frameCount < 0) {
        continue;
    }

    // Get color and depth frames
    auto colorFrame = frameset->colorFrame();
    auto depthFrame = frameset->depthFrame();
    if(colorFrame != nullptr && depthFrame != nullptr) {
        app.addToRender({ colorFrame, depthFrame });
    }
    if(colorFrame != nullptr) {
        // save the colormap
        if(colorFrame->format() != OB_FORMAT_RGB) {
            if(colorFrame->format() == OB_FORMAT_MJPG) {
                formatConvertFilter.setFormatConvertType(FORMAT_MJPG_TO_RGB888);
            }
            else if(colorFrame->format() == OB_FORMAT_UYVY) {
                formatConvertFilter.setFormatConvertType(FORMAT_UYVY_TO_RGB888);
            }
            else if(colorFrame->format() == OB_FORMAT_YUYV) {
                formatConvertFilter.setFormatConvertType(FORMAT_YUYV_TO_RGB888);
            }
            else {
                std::cout << "Color format is not support!" << std::endl;
                continue;
            }
            colorFrame = formatConvertFilter.process(colorFrame)->as<ob::ColorFrame>();
        }
        formatConvertFilter.setFormatConvertType(FORMAT_RGB888_TO_BGR);
        colorFrame = formatConvertFilter.process(colorFrame)->as<ob::ColorFrame>();
        std::string fileName = rootPath + dirName + "/images/frame_" + formatNumber(frameCount) + ".jpg";

        auto beginTime = std::chrono::high_resolution_clock::now();
        saveColor(colorFrame, fileName);
        auto endTime = std::chrono::high_resolution_clock::now();
        auto elapsedTime = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - beginTime).count();
        std::cout << "saveColor processing time: " << elapsedTime << " milliseconds" << std::endl;
        colorCount++;
    }
    else {
        std::cout << "Color frame is null!" << std::endl;
    }

    if(depthFrame != nullptr) {
        // save the depth map
        std::string fileName = rootPath + dirName + "/depth/depth_frame_" + formatNumber(frameCount) + ".png";
        saveDepth(depthFrame, fileName);
        depthCount++;
    }
    else {
        std::cout << "Depth frame is null!" << std::endl;
    }

    auto currentTime = std::chrono::high_resolution_clock::now();
    auto elapsedTime = std::chrono::duration_cast<std::chrono::seconds>(currentTime - startTime).count();
    if (elapsedTime >= 1) {
        float fps = (frameCount - lastFrameCount) / elapsedTime;
        std::cout << "Frame Rate: " << fps << " FPS" << std::endl;

        // Reset the start time and frame count
        startTime = currentTime;
        lastFrameCount = frameCount;
    }
}

` However the print-out FPS is only 10 frame per second. And "Color frame is null!" message is frequently outputted. The viewer runs not smoothly as before. I check the cv2.imwrite speed. The saving process of color image takes up about 25-35 ms. I also tried asynchronously obtain the color stream but still has low frame rate. Did I configure wrongly in some places? Can I inquire is there any good way to save RGBD datastream to disk (like in png format or mp4 format, I possibly need raw color and depth image). If you can help, many thanks! Operating System: Ubuntu 22.04 GPU version: NVIDIA GeForce RTX 4060

[zhonghong322]

I looked at your code for saving images and the logic behind it, you are capturing Color at a 4K resolution and then using D2C, which sets the depth resolution to also be 4K, matching the Color resolution.

Both Depth and Color data are at a 4K resolution, and your image saving logic is on the main thread, which can lead to frame drops. When using the SDK to retrieve a Frameset, quick processing is essential to promptly release the SDK's memory. Internally, the SDK operates on a queue, and if the application processes data slowly, the SDK might drop frames. Suggestions:

1、Implement a separate thread for image saving logic. 2、Writing such high-resolution data to disk is likely to be a bottleneck， first, test the writing speed to the disk.

[bowieshi]

Thank you for your reply. I wrote a multithreading code and use smaller resolution the problem solved. Now with resolution 1920 * 1440, the save to disk operation can work on 30 fps.