How are callbacks internally implemented?

[sidy3d]

Required Info
Camera Model	D455
Firmware Version	5.16.0.1
Operating System & Version	Windows 11
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC
SDK Version	Intel® RealSense™ SDK 2.0 (v2.55.1) }
Language	C++
Segment	other

Issue Description

Hi,

I understand that callbacks (described here) are not to be used for lengthy processing but what happens when they do not return fast enough (yes, I know that rs2::frame_queue exist to ease processing on another thread). Other camera libraries I’ve used document the behavior of callbacks though I can't find specific details for RealSense. Could you shed light on this?

Example: Assume that streaming is done from a device at 90fps without synching and each sensor has its own callback. To avoid frame drops the callback should return in <11ms. Now assume that a single callback run took 500ms to run due to system load.

Questions:

1. As new frames arrive are they circularly buffered internally using RS2_OPTION_FRAMES_QUEUE_SIZE? Do new frames get dropped when the buffer is full or does the oldest unclaimed frame get replaced?
2. Is it the case that only one callback is ever active at a time per stream? As new frames arrive for a sensor will the callbacks be called even though it hasn't returned from the previous call, or will they wait until the callback returns?
3. Does the answer to the above also apply to the IMU data?
4. I see that the info log provides messages in the form Frame Callback Color #<num> overdue, can RS2_OPTION_TOTAL_FRAME_DROPS be used to reliably check for frame drops?

Thank you for the help.

[MartyG-RealSense]

Hi @sidy3d Thanks very much for your questions.

1. The frame queue can hold 16 frames of each stream type by default. The queue size can be increased to hold more frames simultaneously, at the expense of increased memory usage. The frame queue is like a continuously moving conveyor belt, with new frames arriving at the start of the conveyor and the oldest frames dropping off the queue at the end of it.

2. The documentation of the rs-callback C++ callback example program at the link below confirms that the callback can be called simultaneously from multiple sensors.

https://github.com/IntelRealSense/librealsense/tree/master/examples/callback

There is not information available though regarding whether or not callbacks wait until return, unfortunately.

3. https://github.com/IntelRealSense/librealsense/issues/6424 discusses callbacks in regard to IMU data.

4. I would consider the current FPS to be a better indicator of reduction in performance than frame drops.

[sidy3d]

Hi @MartyG-RealSense thank you very much for your response. To re-clarify my questions:

1. When the internal frame queue, controlled by RS2_OPTION_FRAMES_QUEUE_SIZE is full and the user is still holding on to all frames can we assume that new frames are dropped until the user has released the oldest frame?
2. In my use case there is a single callback per sensor, which is different than the linked example that uses a single callback for multiple sensors. In case would only a single callback per sensor ever be active at a time, ex. if the callback for the depth sensor is slow then it won't be called again until it has returned?
3. Good information is this that link but it doesn't really address my question.
4. My understanding is that RS2_OPTION_TOTAL_FRAME_DROPS is used to track cases when the user does not release frames fast enough. Can you elaborate more on this answer of why using FPS is better?

I would appreciate answer from a developer on these questions.

Thanks,

[MartyG-RealSense]

1. When the frame queue is full it becomes 'jammed up', which can lead to warnings such as 'No Frames Received' and 'Frame didn't arrive within 5000' (no new frames during the last 5 seconds). Streaming stops once the queue is full until frame resources are released.

2. I will refer this question to my Intel RealSense colleagues. Thanks very much for your patience.

3. If you look at the frame drop graph in the RealSense Viewer tool, the line representing the drop rate can appear to be peaking continuously at the top of the graph whilst FPS remains high and stable. So when FPS is high and stable, the impression of poor performance given by the frame drop graph's peaks can be misleading.

[sidy3d]

Thanks for the answers. I look forward to the answer to questions 2. and 3.

[MartyG-RealSense]

Hi @sidy3d My Intel RealSense colleagues responded that there is "one callback per sensor. It gets calls whenever a frame becomes available and will wait until it returns. This is the same case for IMU".

[sidy3d]

Hello @MartyG-RealSense, that you very much for the reply. That has answered all of my questions, I will close this issue no.

[MartyG-RealSense]

You are very welcome, @sidy3d - I'm pleased that I could help. :)