[BUG] Camera capture delay between RGB images and disparity images.

LuoQuestionmark commented 1 year ago

Describe the bug

I am working on a program that uses RGB images and disparity images at the same time, using a OAK Pro camera via USB connection. I have noticed that, with my current setting, the output of the disparity images can be delayed. This delay is noticeable with the use of function ImgFrame::getTimestampDevice and can be confirmed visually.

Minimal Reproducible Example

#include "depthai/depthai.hpp"
#include <mutex>
#include <thread>

typedef dai::node::ColorCamera ColorCamera;
typedef dai::node::MonoCamera MonoCamera;
typedef dai::node::StereoDepth StereoDepth;
typedef dai::node::XLinkOut XLinkOut;

typedef std::chrono::time_point<std::chrono::steady_clock, std::chrono::steady_clock::duration> time_point;

namespace {
std::mutex mutex;
cv::Mat rgb_frame, depth_frame;
time_point rgb_point, depth_point;
bool isRunning;
} // namespace

int main() {
    dai::Pipeline pipeline;
    std::shared_ptr<ColorCamera> rgb_cam;
    std::shared_ptr<XLinkOut> rgb_xout, depth_xout;
    std::shared_ptr<MonoCamera> mono_cam_left, mono_cam_right;
    std::shared_ptr<StereoDepth> depth_cam;

    // RGB
    rgb_cam = pipeline.create<ColorCamera>();
    rgb_cam->setBoardSocket(dai::CameraBoardSocket::CAM_A);
    rgb_cam->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    rgb_cam->setIspScale(1, 3);
    rgb_cam->setFps(24);
    rgb_cam->setIsp3aFps(12);

    rgb_xout = pipeline.create<XLinkOut>();
    rgb_xout->setStreamName("RGB");
    rgb_xout->input.setQueueSize(1);
    rgb_xout->input.setBlocking(false);

    rgb_cam->video.link(rgb_xout->input);

    // MONO (* 2)
    mono_cam_left = pipeline.create<MonoCamera>();
    mono_cam_left->setBoardSocket(dai::CameraBoardSocket::CAM_B);
    mono_cam_left->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
    mono_cam_left->setNumFramesPool(1);
    mono_cam_left->setFps(24);
    mono_cam_left->setIsp3aFps(12);

    mono_cam_right = pipeline.create<MonoCamera>();
    mono_cam_right->setBoardSocket(dai::CameraBoardSocket::CAM_C); // left mono camera
    mono_cam_right->setNumFramesPool(1);
    mono_cam_right->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
    mono_cam_right->setFps(24);

    // DEPTH
    depth_cam = pipeline.create<dai::node::StereoDepth>();
    depth_cam->setLeftRightCheck(true);
    depth_cam->setSubpixel(true);
    depth_cam->setSubpixelFractionalBits(4);
    depth_cam->setDefaultProfilePreset(dai::node::StereoDepth::PresetMode::HIGH_DENSITY);
    depth_cam->initialConfig.setMedianFilter(dai::MedianFilter::MEDIAN_OFF);

    depth_xout = pipeline.create<dai::node::XLinkOut>();
    depth_xout->setStreamName("DEPTH");
    depth_xout->input.setQueueSize(1);
    depth_xout->input.setBlocking(false);

    mono_cam_left->out.link(depth_cam->left);
    mono_cam_right->out.link(depth_cam->right);
    depth_cam->disparity.link(depth_xout->input);

    isRunning          = true;
    std::thread update = std::thread([pipeline]() {
        while (isRunning) {
            try {
                dai::Device device(pipeline);
                std::shared_ptr<dai::DataOutputQueue> rgb_queue   = device.getOutputQueue("RGB");
                std::shared_ptr<dai::DataOutputQueue> depth_queue = device.getOutputQueue("DEPTH");

                while (isRunning) {
                    std::shared_ptr<dai::ImgFrame> rgb_packet   = rgb_queue->get<dai::ImgFrame>();
                    std::shared_ptr<dai::ImgFrame> depth_packet = depth_queue->get<dai::ImgFrame>();

                    {
                        std::unique_lock<std::mutex> lock(mutex);
                        rgb_frame   = rgb_packet->getCvFrame();
                        depth_frame = depth_packet->getCvFrame();
                        rgb_point   = rgb_packet->getTimestampDevice();
                        depth_point = depth_packet->getTimestampDevice();
                    }
                }
            } catch (const std::runtime_error &error) {
                puts(error.what());
                std::this_thread::sleep_for(std::chrono::milliseconds(500));
            }
        }
    });

    while (isRunning) {
        {
            {
                std::unique_lock<std::mutex> lock(mutex);
                std::cout << (rgb_point - depth_point).count() << std::endl;
            }
            std::this_thread::sleep_for(std::chrono::seconds(1));
        }
    }

    update.join();
}

Expected behavior

The delay measured and printed as output is increasing during the execution of program. I write a script to visualize the delay.

Screenshots

x-axis: Time since program launched (seconds); y-axis: Delay between two cameras (nanoseconds).

Pipeline graph

Similar to the previous bug report shown here. Except no monochromatic output.

283807921-83f929d1-fbac-4bcc-9952-7d7a2d2ff110

Attach system log

{
    "architecture": "64bit ELF",
    "machine": "x86_64",
    "platform": "Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35",
    "processor": "x86_64",
    "python_build": "main Jun 11 2023 05:26:28",
    "python_compiler": "GCC 11.4.0",
    "python_implementation": "CPython",
    "python_version": "3.10.12",
    "release": "5.15.133.1-microsoft-standard-WSL2",
    "system": "Linux",
    "version": "#1 SMP Thu Oct 5 21:02:42 UTC 2023",
    "win32_ver": "",
    "packages": [
        "anyio==4.0.0",
        "apriltag==0.0.16",
        "argon2-cffi==23.1.0",
        "argon2-cffi-bindings==21.2.0",
        "arrow==1.3.0",
        "astroid==2.9.3",
        "asttokens==2.4.0",
        "async-lru==2.0.4",
        "attrs==23.1.0",
        "Babel==2.13.1",
        "backcall==0.2.0",
        "beautifulsoup4==4.12.2",
        "bleach==6.1.0",
        "blinker==1.4",
        "certifi==2023.7.22",
        "cffi==1.16.0",
        "charset-normalizer==3.3.2",
        "comm==0.1.4",
        "command-not-found==0.3",
        "contourpy==1.1.0",
        "cryptography==3.4.8",
        "cycler==0.11.0",
        "dbus-python==1.2.18",
        "debugpy==1.8.0",
        "decorator==5.1.1",
        "defusedxml==0.7.1",
        "depthai==2.20.2.0",
        "depthai-pipeline-graph @ git+https://github.com/luxonis/depthai_pipeline_graph.git@848a147009ffca578a85910ed298ab4430c67727",
        "distro==1.7.0",
        "distro-info==1.1+ubuntu0.1",
        "dot2tex==2.11.3",
        "entrypoints==0.4",
        "exceptiongroup==1.1.3",
        "executing==2.0.0",
        "fastjsonschema==2.19.0",
        "fonttools==4.40.0",
        "fqdn==1.5.1",
        "gprof2dot==2022.7.29",
        "greenlet==2.0.2",
        "httplib2==0.20.2",
        "idna==3.4",
        "importlib-metadata==4.6.4",
        "ipykernel==6.25.2",
        "ipython==8.16.1",
        "isoduration==20.11.0",
        "isort==5.6.4",
        "jedi==0.19.1",
        "jeepney==0.7.1",
        "Jinja2==3.1.2",
        "json5==0.9.14",
        "jsonpointer==2.4",
        "jsonschema==4.20.0",
        "jsonschema-specifications==2023.11.1",
        "jupyter-events==0.9.0",
        "jupyter-lsp==2.2.0",
        "jupyter_client==7.4.9",
        "jupyter_core==5.4.0",
        "jupyter_server==2.10.1",
        "jupyter_server_terminals==0.4.4",
        "jupyterlab==4.0.8",
        "jupyterlab-pygments==0.2.2",
        "jupyterlab_server==2.25.1",
        "keyring==23.5.0",
        "kiwisolver==1.4.4",
        "launchpadlib==1.10.16",
        "lazr.restfulclient==0.14.4",
        "lazr.uri==1.0.6",
        "lazy-object-proxy==0.0.0",
        "logilab-common==1.8.2",
        "MarkupSafe==2.1.3",
        "matplotlib==3.7.1",
        "matplotlib-inline==0.1.6",
        "mccabe==0.6.1",
        "mistune==3.0.2",
        "more-itertools==8.10.0",
        "msgpack==1.0.5",
        "mypy-extensions==0.4.3",
        "nbclient==0.9.0",
        "nbconvert==7.11.0",
        "nbformat==5.9.2",
        "neovim==0.3.1",
        "nest-asyncio==1.5.8",
        "netifaces==0.11.0",
        "notebook==7.0.6",
        "notebook_shim==0.2.3",
        "numpy==1.24.2",
        "oauthlib==3.2.0",
        "opencv-contrib-python==4.8.0.74",
        "overrides==7.4.0",
        "packaging==23.1",
        "pandas==1.5.3",
        "pandocfilters==1.5.0",
        "parso==0.8.3",
        "pexpect==4.8.0",
        "pickleshare==0.7.5",
        "Pillow==10.0.0",
        "pip==23.2.1",
        "platformdirs==2.5.1",
        "prometheus-client==0.18.0",
        "prompt-toolkit==3.0.39",
        "psutil==5.9.5",
        "ptyprocess==0.7.0",
        "pure-eval==0.2.2",
        "pycparser==2.21",
        "Pygments==2.11.2",
        "PyGObject==3.42.1",
        "PyJWT==2.3.0",
        "pylint==2.12.2",
        "pynvim==0.4.3",
        "pyparsing==2.4.7",
        "PyQt5==5.15.6",
        "PyQt5-sip==12.9.1",
        "python-apt==2.4.0+ubuntu2",
        "python-dateutil==2.8.2",
        "python-json-logger==2.0.7",
        "pytz==2023.3",
        "PyYAML==5.4.1",
        "pyzmq==24.0.1",
        "Qt.py==1.3.8",
        "referencing==0.31.0",
        "requests==2.31.0",
        "rfc3339-validator==0.1.4",
        "rfc3986-validator==0.1.1",
        "rpds-py==0.13.0",
        "scipy==1.11.1",
        "SecretStorage==3.3.1",
        "Send2Trash==1.8.2",
        "setuptools==59.6.0",
        "six==1.16.0",
        "sniffio==1.3.0",
        "soupsieve==2.5",
        "ssh-import-id==5.11",
        "stack-data==0.6.3",
        "systemd-python==234",
        "terminado==0.18.0",
        "tinycss2==1.2.1",
        "toml==0.10.2",
        "tomli==2.0.1",
        "tornado==6.3.3",
        "traitlets==5.11.2",
        "types-pyside2==5.15.2.1.6",
        "types-python-dateutil==2.8.19.14",
        "typing_extensions==4.8.0",
        "ubuntu-advantage-tools==8001",
        "ufw==0.36.1",
        "unattended-upgrades==0.1",
        "uri-template==1.3.0",
        "urllib3==2.1.0",
        "wadllib==1.3.6",
        "wcwidth==0.2.8",
        "webcolors==1.13",
        "webencodings==0.5.1",
        "websocket-client==1.6.4",
        "wheel==0.37.1",
        "wrapt==1.13.3",
        "yappi==1.4.0",
        "zipp==1.0.0"
    ],
    "usb": [
        "NoLib"
    ],
    "uname": [
        "Linux DESKTOP-FI4JTJE 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64"
    ]
}

LuoQuestionmark commented 1 year ago

Hello, I have done two more tests, it seems that the problem happens, in general, quite randomly. The program can be working for hours, but then the delay kicks in and it cannot recover from that by itself.

Example that it fails after hours:

delay2

Example that it works:

delay3

In our application it will be troublesome to restart the system, which, can reset the synchronization. Is there any other approach to deal with this problem?

LuoQuestionmark commented 11 months ago

While the cause of the increasing delay is still obscure, I have the impression that the delay is monotonic increasing is from a bug from the 2.22 version of the library; by updating to the newest version 2.23 the delay seems to be more reasonable. Each time the system reconnect because of a bad connectivity, the delay goes back to a reasonable small value. (While the reason of bad connectivity is yet another problem.

luxonis / depthai