Documentation or example on using the rs2::asynchronous_syncer

[yashmanian]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D400
Operating System & Version	Ubuntu 16
Platform	PC
Language	C++

Issue Description

I am trying to asynchronously get access to frames from the camera. I am defining both the depth and color sensor separately, and I am using the sensor.start() callback to pass in an object that will asynchronously grab frames for me. The problem is that I am unable to get the color and depth frames at the same time. Meaning the frame index on the color frame will be one behind the depth sensor. I was previously told that I could use the rs2::asynchronous_syncer to pass in sensor objects, and it will get me a synced frameset via asynchronous callback. The post can be found here: https://github.com/IntelRealSense/librealsense/issues/2647 I have been unable to find any documentation on using this function to sync frames. It would be great if someone could point me in the direction of more descriptive documentation, suggest an alternative to my problem.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @yashmanian,

Still need any support for this topic?

[yashmanian]

I have a solution working, but I would like to learn about the rs2::asynchronous_syncer. I still haven't found any good documentation about this. It would be great if you could point me in the direction of some

[dorodnic]

I agree we still need to improve the documentation regarding the syncer.

There are many aspects to this topic:

Hardware

• Fist things first, before any of this can work properly, UVC metadata must be enabled. Otherwise both frame number and the timestamp are mere approximations. You can ensure this is the case by checking timestamp_domain of frames, it should be HARDWARE_CLOCK.
• Next, if your goal is visual sync between color and depth, hardware sync is only enabled when the two sensors are running at the same actual FPS. Since auto-exposure by default can reduce FPS, it is recommended to disable Exposure Priority option on the color sensor.
• Timestamps of synced color and depth frames will not be the same because the two sensors assign timestamp at different times during frame exposure (I believe color is mid-exposure and depth is start-exposure, but might be the other way around)
• Frame-counter is not always reliable. Based on the feedback we got from firmware and hardware teams, the only real use for the frame number can be frame-drops detection. It is unsuitable for matching between devices.
• Within the stereo-module device, we can rely on the frame-counter because it is assigned at once to depth and the corresponding infrared frame.
• Between depth and color, the best heuristic we can offer is matching based on closest timestamps. While we don't know exactly the delta between color and depth timestamps, we can assume it is less then 0.5 / FPS, ensuring that nearest match is the right one.

syncer class

• syncer object is designed to abstract all these hardware details and just do the right thing for whatever device you throw at it. It will get the device from input frames, and ask the device for the preferred synchronization scheme
• The only real method to test syncer performance is by recording object in motion, and visually examining the object in matched frames.
• Since most people prefer to get frames synchronously, the default syncer class exposes blocking wait_for_frames API.
• If you want the results asynchronously, that's where asynchronous_syncer comes into play (I know, terrible name)
• The design of the syncer class ensures that it will do all the matching on the original callback thread, and it will do it (relatively) quickly.

Frame-set expansion

• In some situations, the syncer can decide to output a single frame when it knows no match is going to be found. (one example is when the match was lost due to frame-drop, another is non-matching frame-rates) This makes working with the syncer a bit tricky, because you need to always check if you got a match or a single frame.
• Customer feedback showed very early that people strongly prefer to always get a full frameset, even if sometimes it is not best synchronized. This is what pipeline is doing. In case of no-match it will expand the frameset with older frames. Pipeline is adding an extra processing block that was recently renamed to aggregator for this exact purpose.

Apologize for the delayed response, let me know if you have any questions.