search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.61k stars 4.83k forks source link

Intermittent delay on arrival of first RGB frame when using intercam sync #11203

Open RobbertProost opened 1 year ago

RobbertProost commented 1 year ago
Required Info
Camera Model 2x D415
Firmware Version 5.13.0.50
Operating System & Version Linux
Kernel Version (Linux Only) 5.4.181
Platform PC
SDK Version 2.50
Language C++
Segment {Robot/Smartphone/VR/AR/others }

Issue Description

For our product we are using two RealSense D415 camera's. To synchronize the the images of both camera's we use the HW sync, where we connected the two camera's as described in the multicam white paper .

Now we are seeing some weird intermittent delay in the arrival of the first frames of the RGB stream of the camera that is configured as master when enabling the Depth and the RGB stream. What we observe is that the time between enabling the RGB stream and receiving the first frame is not stable and sometimes takes longer than 5 seconds. In that scenario the RealSense SDK also logs the statement Frames didn't arrived within 5 seconds.

As soon as we change the inter cam sync back to default (value '0') we no longer see this problem. Next to that we found out that problem only seems to occur when the frame-rate of the Depth sensor and the RGB sensor are in sync. If they are not in sync (e.g. Depth = 15FPS, RGB = 30FPS), the problem doesn't occur.

So the problem seems to be related to the sync master and the fact that the Depth and the RGB sensor are running at the same frame-rate.

It can be reproduced by using the realsense-viewer application and only having a single camera (but with sync master enabled). Steps to reproduce:

Based on these observation we have a strong feeling that there might be a problem in the firmware of the camera, causing some internal trigger issues when FPS are in sync and inter cam sync master is enabled.

Is this a known issue or maybe expected behavior? Or are we using it wrong?

RobbertProost commented 1 year ago

This issue is also reproducible on Windows 11 with the realsense-viewer.exe application.

MartyG-RealSense commented 1 year ago

Hi @RobbertProost In the multicam white paper you linked to that covers Inter Cam Sync Mode 1 (Master) and 2 (Slave), only the depth stream of the slave camera is timestamp-synced with the master camera. The RGB stream is not synced.

Inter Cam Sync Mode is not responsible for syncing depth and RGB sensors on the same camera when a single camera's Mode is set to '1'.

Depth-RGB sync on the same camera should kick in when both streams have the same FPS value. If auto-exposure is enabled and an RGB option called Auto-Exposure Priority is disabled then the RealSense SDK will aid depth-RGB sync by enforcing a constant FPS for the two streams instead of permitting the FPS of one of the streams to vary, as confirmed by a RealSense team member at https://github.com/IntelRealSense/librealsense/issues/5675

I repeated your single-camera test with my own D415 in SDK 2.53.1 and firmware 5.14.0.0. I observed that whilst there was a perceptible delay when starting the RGB stream when Inter Cam Sync Mode was set to '1' compared to a faster response with '0', it was mostly around 1 second delay (sometimes 2 seconds) rather than the 5 that you experienced. On the occasions when the delay was 2 seconds, a 'No Frames Received' warning would briefly be displayed.

RobbertProost commented 1 year ago

It is clear to us that the HW sync between the two camera's is only impacting the depth stream. However we still want to be able to also capture RGB frames from the cameras while also capturing depth images. The problem is the nondeterministic behavior of the RGB stream. In our application we occasionally start and stop the camera stream and to verify that a stream is correctly started we wait for the first frame to arrive. Currently the timeout is set to 5 seconds (which is already pretty long), and we still occasionally see that the RGB stream is not producing frames yet.

Good to hear that you observe similar behavior as we do. For us it is also most of the time somewhere between 1 and 3 seconds before the first frames arrive, but occasionally it takes longer than 5 seconds. With a small test application that configures the camera, start the depth stream and continuously disable->enable the RGB stream in a loop results in reproducing a delay of 5 seconds or more within 2 to 30 minutes of running.

Our suspicion is that in the camera itself there is some conflict between internally syncing the depth and RGB sensor when the frame-rates match and the depth sensor acting as a sync master for the slave cameras. Could you check with the RealSense team if this is the case and if anything can be done about it?

MartyG-RealSense commented 1 year ago

I have highlighted this issue to my Intel RealSense colleagues as you suggested.

MartyG-RealSense commented 1 year ago

Hi again @RobbertProost Thanks very much for your patience. One of my Intel RealSense colleagues have responded with information regarding your issue.

Firstly, they thought that a 5 second delay sounded too long. Like in my own tests, the delay that they typically experienced was 1-2 seconds. They were testing D415 with Windows rather than Linux though. My tests were also run in Windows. So it raises the possibility that a 5 second delay may be a problem specific to Linux.

My colleague requested whether you would be able to provide your testing tool app so that they can perform the same test that you did and check what the delay looks like.

They also provided the following further information.

  1. The RGB sensor itself only outputs one format – the full size 1920x1080. And the Image Signal Processor (ISP) is doing the job of scaling / cropping.
  2. Although the Depth sensor and RGB sensor are the same chip in the D415 camera model, their configuration / settings are different.
  3. Intel recommend to use the same frame rate for both Depth and RGB. It is noted though that it you were experiencing less delay while running at different frame rates.
MartyG-RealSense commented 1 year ago

Hi @RobbertProost Was the advice in the comment above helpful, please? Thanks!

RobbertProost commented 1 year ago

Sorry for not responding earlier. Thanks for the feedback and the additional insights. This however does not solve our issue. Although it is a bit tedious to replicate the behavior with the RealSense-Viewer it does show itself occasionally. Just yesterday I encountered it with the RealSense-Viewer on Linux where it even took almost 10 seconds before the first frame arrived.

Let me get back to you with a small test application that I can share that triggers it automatically. Currently my test application includes a dependency on our internal logging framework, which I cannot share.

MartyG-RealSense commented 1 year ago

Thanks very much @RobbertProost for the update. I look forward to your next report.

RobbertProost commented 1 year ago

Good afternoon @MartyG-RealSense. I found the time to make a small application that reproduces this issue with only using the realsense SDK as dependency.

The application configures the bare minimum:

After that it opens the depth sensor and starts the depth stream at a resolution of 1280x720 and frame rate of 15FPS. Then the while loop starts where we measure how long it takes to open the color sensor, start the stream and wait for the first frame to arrive.

One immediate observation is the variety in times. The majority of times it takes a little less than 700ms, but there are a lot of spikes any where between 1 to 8! seconds. While running this application I typically measure a start time of more than 5 seconds within 15 minutes of running.

If I run the same test application but set the intercam sync setting to '0' (default), the start time is dead stable and always around 750-760ms.

Hopefully this helps with the investigation into what is going on here.

#include <chrono>
#include <iostream>
#include <librealsense2/rs.hpp>
#include <sstream>
#include <thread>

rs2::stream_profile getStreamProfile(rs2::sensor sensor, int width, int height, int fps) {
  for (auto p : sensor.get_stream_profiles()) {
    if (p.as<rs2::video_stream_profile>().width() == width && p.as<rs2::video_stream_profile>().height() == height &&
        p.fps() == fps) {
      return p;
    }
  }
  return rs2::stream_profile{};
}

int main(int /*argc*/, char** /*argv[]*/) try {
  rs2::context ctx;

  auto devices = ctx.query_devices();
  auto device_count = devices.size();
  if (device_count == 0) {
    std::cout << "No device detected. Is it plugged in?\n";
    return EXIT_SUCCESS;
  }
  // Get the first connected device
  auto dev = devices[0];

  auto depth_sensor = dev.first<rs2::depth_sensor>();
  auto color_sensor = dev.first<rs2::color_sensor>();
  depth_sensor.set_option(RS2_OPTION_INTER_CAM_SYNC_MODE, 1);
  color_sensor.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0);

  rs2::syncer sync;

  depth_sensor.open(getStreamProfile(depth_sensor, 1280, 720, 15));
  depth_sensor.start(sync);

  while (true) {
    auto start_time = std::chrono::system_clock::now();
    color_sensor.open(getStreamProfile(color_sensor, 1920, 1080, 15));
    color_sensor.start(sync);

    rs2::frameset frame_set;
    do {
      std::this_thread::sleep_for(std::chrono::milliseconds(10));
      sync.poll_for_frames(&frame_set);
    } while (!frame_set.first_or_default(rs2_stream::RS2_STREAM_COLOR));

    auto duration = std::chrono::system_clock::now() - start_time;
    std::cout << "Enabling the color stream took: "
              << std::chrono::duration_cast<std::chrono::milliseconds>(duration).count() << std::endl;

    color_sensor.stop();
    color_sensor.close();
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
  }

  return EXIT_SUCCESS;
} catch (const rs2::error& e) {
  std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    "
            << e.what() << std::endl;
  return EXIT_FAILURE;
} catch (const std::exception& e) {
  std::cerr << e.what() << std::endl;
  return EXIT_FAILURE;
}
MartyG-RealSense commented 1 year ago

Thanks very much @RobbertProost for your test script. I have passed it to my colleague who provided the earlier advice at https://github.com/IntelRealSense/librealsense/issues/11203#issuecomment-1367157783

MartyG-RealSense commented 1 year ago

I wanted to update you that testing of your issue by my Intel RealSense colleagues is ongoing. In their most recent test, they ran the test script for 360 cycles and the longest delay that occurred was 2.44 seconds.

MartyG-RealSense commented 1 year ago

Hi @RobbertProost Do you have an update about this case that you can provide, please? Thanks!

MartyG-RealSense commented 1 year ago

Hi @RobbertProost Do you require further assistance with this case, please? Thanks!

RobbertProost commented 1 year ago

Hi @MartyG-RealSense I am actually waiting on feedback from your side on the possible root-cause based on the tests your team did with my sample application. Can you confirm if you also notice the inconsistency in the timing of the receiving the first frame after enabling the sensor? And if not, can you indicate on what platform (Windows or Linux, kernel version, etc) these tests have been performed such that we can start excluding environmental factors.

MartyG-RealSense commented 1 year ago

I have passed your latest questions to my Intel RealSense colleagues who performed the testing, Thanks again!

MartyG-RealSense commented 1 year ago

Hi again @RobbertProost My colleagues have responded to your questions.

Processor: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz 3.00 GHz Installed RAM: 32.0 GB (31.7 GB usable) System type: 64-bit operating system, x64-based processor

MartyG-RealSense commented 1 year ago

Hi @RobbertProost Was the information in the comment above that was provided by my Intel RealSense colleagues helpful to you, please? Thanks!

RobbertProost commented 1 year ago

I have tested with multiple camera's, all showing the same behavior. I am still curious if the time from start to first frame is stable in the tests that your colleague performed, or if that is also fluctuating like in our case. Even if in your tests the max time doesn't exceed 3 seconds, I still think its worth figuring out why the time till the first frame is not stable (compared to the situation where the camera is not a sync master).

MartyG-RealSense commented 1 year ago

Hi @RobbertProost I have directed your latest query to my colleague.

MartyG-RealSense commented 1 year ago

My colleague is monitoring USB traffic to see how long it takes from the start of commanding to the start of video streaming on the camera side. I will update you when I receive feedback from these tests. Thanks again for your patience!

RobbertProost commented 1 year ago

Thanks, I'm looking forward to the results.

MartyG-RealSense commented 1 year ago

During tests of USB traffic, my colleague confirmed that the D415 RGB needs some time to settle down the video while running as a slave cam. The investigation will continue.

RobbertProost commented 1 year ago

Thanks for the update. Just to make sure we are investigating the correct thing: you mention 'while running as slave cam', while my behavior is showing when my camera is configured as 'master cam'. Did you also mean 'master cam'?

MartyG-RealSense commented 1 year ago

I will ask my colleague to test for whether the delay also occurs on the master cam.

MartyG-RealSense commented 1 year ago

Hi @RobbertProost Thanks very much for your patience. My colleague re-tested with the master cam and found that the RGB delay results were the same as their earlier slave cam test.

RobbertProost commented 1 year ago

Ok. Is there already any insight in the possible root-cause?

MartyG-RealSense commented 1 year ago

My colleague theorized that it may be related to the camera firmware driver but we do not have a means to confirm that theory. They were not able to replicate your 5 second delay during tests, with the worst result being 2.317 seconds.

MartyG-RealSense commented 1 year ago

Hi @RobbertProost Do you have an update about this case that you can provide, please? Thanks!

RobbertProost commented 1 year ago

Hi Marty, I repeated the experiment on a Windows machine just to double check that the underlying OS doesn't impact the result and I see more or less the same values. Similar to your colleagues results, most of the timings are between 700ms and 2300ms, but when running long enough I do also get measurements as extreme as 7100ms.

Below results are created with a Windows 11 laptop with a D415 camera connected via USB-C.

image

The theory of your collegue about that it is potentially caused by the camera firmware was also my initial thought (as can be read in my opening post). I am not sure why you say that 'you don't have means to confirm this theory'. Don't you have the possibilities to debug the firmware of the camera?

MartyG-RealSense commented 1 year ago

The support department of RealSense is not able to access the contents of the firmware driver to perform adjustments and tests on it.

RobbertProost commented 1 year ago

Ok, clear. How do we get this issue on the radar of the development team?

MartyG-RealSense commented 1 year ago

I have passed your request to my colleagues.

MartyG-RealSense commented 1 year ago

Hi @RobbertProost An official Intel internal bug report has been created for this D415 RGB master cam delay and I have added an Enhancement label to this page to indicate that the issue should be kept open whilst the internal bug report is active. Thanks again for your patience!

RobbertProost commented 1 year ago

Thank you very much. Keep me posted!

RobbertProost commented 1 year ago

Hi @MartyG-RealSense, can you give an update on the status of the internal bug report?

MartyG-RealSense commented 1 year ago

Hi @RobbertProost I am not able to share details of internal reports, unfortunately. The report that you are asking about remains open and work continues on it.