Left-IR (RGB8) and depth not in sync

[rafaelspring]

Key	Value
Camera Model	D415
Firmware Version	05.09.11.00
Operating System & Version	Win10
Platform	PC
SDK Version	2.11.1

The D415 left IR image and the depth image are not in sync. This was tested with left IR being set to 1280x720 RGB8 at 30 fps.

Assuming frames are queried fast enough the depth lags behind the IR by exactly one frame. If frames are not queried fast enough the lag varies. Here are some timestamp pairs (depth, color):

772438 772471
772471 772504
772504 772538
772538 772571
772571 772604
772604 772638
772638 772671
772671 772704
772704 772738
772738 772771
772771 772804
...

Here's the code that generated these. This is based on the rs-capture example:

// License: Apache 2.0. See LICENSE file in root directory.
// Copyright(c) 2017 Intel Corporation. All Rights Reserved.

#include <iostream>

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include "example.hpp"          // Include short list of convenience functions for rendering

// Capture Example demonstrates how to
// capture depth and color video streams and render them to the screen
int main(int argc, char * argv[]) try {
  const int width = 1280;
  const int height = 720;

  // Create a simple OpenGL window for rendering:
  window app(width, height, "RealSense Capture Example");
  // Declare two textures on the GPU, one for color and one for depth
  texture depth_image, color_image;

  // Declare depth colorizer for pretty visualization of depth data
  rs2::colorizer color_map;

  // Declare RealSense pipeline, encapsulating the actual device and sensors
  rs2::pipeline pipe;

  rs2::config cfg;
  cfg.enable_stream(RS2_STREAM_INFRARED, 0, width, height, RS2_FORMAT_RGB8, 30);
  cfg.enable_stream(RS2_STREAM_DEPTH, -1, width, height, RS2_FORMAT_Z16, 30);
  cfg.disable_stream(RS2_STREAM_COLOR);

  // Start streaming with default recommended configuration
  pipe.start(cfg);

  while(app) // Application still alive?
  {
    rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera

    rs2::frame depth = color_map(data.get_depth_frame()); // Find and colorize the depth data
    rs2::frame color = data.get_color_frame();            // Find the color data

    std::cout << data.get_depth_frame().get_timestamp() << " "
                << color.get_timestamp() << std::endl;

    // Render depth on to the first half of the screen and color on to the second
    depth_image.render(depth, { 0,               0, app.width() / 2, app.height() });
    if (color) {
      color_image.render(color, { app.width() / 2, 0, app.width() / 2, app.height() });
    }
  }

  return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
  std::cerr << "RealSense error calling " << e.get_failed_function() << "("
          << e.get_failed_args() << "):\n    " << e.what() << std::endl;
  return EXIT_FAILURE;
}
catch (const std::exception& e)
{
  std::cerr << e.what() << std::endl;
  return EXIT_FAILURE;
}

[dorodnic]

I suspect the frame metadata isn't available. We can check using the Viewer, by clicking on (i) icon above any of the streams and checking timestamp domain field. It is supposed to be hardware clock.

[rafaelspring]

I don't see a timestamp domain field. I have

• Name
• Serial No
• FW Version
• Min FW Version
• Physical Port
• Debug Op Code
• Advanced Mode
• Product ID
• USB Type Descriptor

I am running version 2.11.1 of the Viewer. Any other way I can check?

[dorodnic]

Please look for (i) icon above the streams themselves (above depth / ir output)

[rafaelspring]

I just checked. It is in fact saying hardware clock for both Depth and Left-camera RGB-8 streams.

[dorodnic]

This should not happen, we will try to reproduce and see what went wrong. @aangerma

[dorodnic]

After initial investigation this seems to be firmware bug, will be resolved in future update.

[rafaelspring]

@dorodnic Is this fixed in any of the newer firmwares?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] @xtrawurst The problem has been fixed in latest FW 5.9.14, and it will be released soon.

[dorodnic]

5.9.14 is now public, please let us know if the issue was resolved.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] @xtrawurst Issue fixed?

[rafaelspring]

@RealSense-Customer-Engineering Yes, 5.9.14 fixed this issue.