Inconsistent frame-rate using external hardware sync on the D455 Cameras

[jasatron9000]

Required Info
Camera Model	D455
Firmware Version	1.15
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.15.0
Platform	PC
Librealsense SDK Version	2.53.1
Language	C++ / Python
Segment
ROS Distro	Noetic
RealSense ROS Wrapper Version	2.3.2

Issue Description

Hi

We are currently running a three camera setup using an external synchronization signal (Genlock) to run all our cameras at a frequency of 10 Hz. We've noticed that some of our cameras' frame-rate has become inconsistent and the frame-rate will typically drop as low as ~2Hz.

The configuration of our multi camera setup is:

• For our external hardware synchronization signal, the pulse width is at 10 ms.
• The depth and color stream are configured to be 1280x720 @ 30fps and 848x480 @ 30 fps respectively.
• The INTER_CAM_SYNC_MODE is set at 4 (Genlock).

Here's a file detailing all the launch file configuration of the cameras: rosparam.txt

We typically run these cameras in an outdoor environment usually for around ~6 hours each day.

We managed to collect some data on this frame-rate issue on two cameras where we did not experience any frame-rate issue and another where we did experience this issue labelled as Good and Bad in the following graphs respectively. To try and isolate the source of this issue, we swapped the interposer, vision processing board and the depth camera module between the good and bad cameras.

The data collection setup was done in 30 minute intervals where only one camera is connected to the PC and these were done in an indoor environment while the cameras were stationary.

The histogram above is the duration between frames using the frame_timestamp field in the depth metadata messages. The good camera's frame duration is consistently at 0.1 seconds ~99% of the time, but for the bad camera's frame duration only sticks to the 0.1 seconds duration between 60-80% of the runtime.

The component that seems to be causing this issue is the depth-camera module.

We've collected data looking at the duration between frames using the timestamp when the bag recorder receives the depth metadata messages. We also observe the same thing here as I've mentioned above.

Would appreciate some input on the cause of this inconsistency with the frame-rate on the camera module.

Thanks in advance.

[MartyG-RealSense]

Hi @jasatron9000 When using genlock mode (Inter Cam Sync Mode 4 and above), the pulse width should be 100 microseconds or more. You stated that your pulse width is 10 ms.

The FPS of the slave cameras should also be set to 2x the FPS of the trigger signal.

The documentation for the genlock feature was removed because it was an experimental proof of concept rather than an official Intel RealSense feature and it was immature and unvalidated. The officially supported, mature and validated method of hardware sync is Mode 1 and 2.

However, genlock mode is still supported within the SDK if you need to use it. You can access an archived PDF document version of the removed documentation at the link below if you have not seen it already.

External Synchronization.pdf

[jasatron9000]

Thanks @MartyG-RealSense for responding.

When using genlock mode (Inter Cam Sync Mode 4 and above), the pulse width should be 100 microseconds or more. You stated that your pulse width is 10 ms.

We set the pulse width to be 10 milliseconds not 100 microseconds. Apologies for the confusion.

---

The FPS of the slave cameras should also be set to 2x the FPS of the trigger signal.

We set the FPS of the slave cameras to 30 FPS which is within the allowable range of the FPS of the trigger signal which is 10 FPS.

---

We want to get some clarification on the depth cameras themselves on why some of the cameras' framerate does not stick to a consistent 10 Hz.

• Is it possible our configuration is improperly set that's causing this difference in performance between the cameras we have?

• You mentioned that the GenLock is immature and unvalidated, could using GenLock lead to some cameras working at a consistent 10 Hz and other cameras not.

• Could this be due to an issue with the hardware itself specifically the depth camera module? We're running these cameras in an outdoor environment so would moisture or temperature cause some of our cameras framerate performance to deteriorate. (We also have some cameras in the same conditions that are still running consistently at 10 Hz)

[MartyG-RealSense]

I cannot see a problem in your ROS launch file.

If you are syncing depth only and using Inter Cam Sync Mode '4' and not one of the more complex higher modes (5 to 260) then I would expect genlock to be stable and relatively reliable.

If you are running the cameras continuously over the 6 hour period then using the officially validated modes (1 and 2) might work fine for you. The key difference between 1 / 2 and 4 are:

• With mode 1 and 2 it is more difficult to use an external signal generator to produce the trigger, as the trigger Hz has to be matched very precisely with the RealSense camera. It would be more typical with mode 1 / 2 to have one camera set to '1' as a Master that generates the trigger pulse, with the other Slave cameras syncing to the Master camera's trigger.

• In modes 1 and 2, the Slave cameras will listen for a trigger on each frame for a certain period of time and then perform an unsynced capture if the trigger is not received. With mode 4, the slaves wait indefinitely for a trigger before performing a capture, and then wait for the next trigger before performing the next capture.

---

Regarding some cameras performing at 10 Hz and others less, if some of the cameras are directly faced towards sunlight then it could cause a drop in exposure value that could result in FPS also dropping. When this occurs, auto-exposure can correct the problem once the camera is no longer directly facing the strong light source. So if the cameras are not able to be moved or rotated then once the camera is disrupted by such light then it might not recover until the sun changes position and is no longer directly facing the camera.

If direct sunlight was affecting some of the cameras then the D455f camera model that is equipped with a light-absorbing filter could handle the light better. If you are not able to change the cameras then you could try purchasing a filter product separately and placing it over the lenses on the front of the camera. If you wish to experiment with this then I would recommend first trying a thin-film linear polarization filter for negating light glare, as these are inexpensive and can be found in stores such as Amazon by searching for linear polarizing filter sheet

I note though that you have tested the cameras indoors and still experienced the problem and so direct sunlight may not be a factor in the issue.

---

In regard to temperature, there may be consequences for the image if the internal temperature of the camera (described by its Asic Temperature or Projector Temperature) goes above around 42 degrees C.

At 60 degrees C and above, the camera firmware driver's safety mechanism will kick in and shut off the projector's laser to attempt to reduce the internal temperature, but this may not help if the external environmental temperature remains very hot. The camera will continue to operate after the laser shuts off but depth image quality may reduce. The laser will also shut off if the internal temperature falls to 0 degrees C or below.

Intel have a D456 camera model that is a D455 with an IP65-rated casing that is protected against entry by dust or water.

[jasatron9000]

We'll give Inter Cam Sync Mode 1 and 2 a shot and see if this will help resolve this issue. Also, would it be possible to run the color and depth streams at 10 FPS while using modes 1 and 2 or can this only be achieved if the external synchronization signals are used.

This might take a bit of time to set up, so I'll get back to you on how the mode 1 and 2 will turn out at some point next week.

---

Regarding your point on temperature, would there be a chance for permanent damage to the depth camera module if temperatures reach above 42 C which may lead to the issues we're seeing here.

[MartyG-RealSense]

High temperature does not usually lead to damage of the camera. There is the possibility that a very high temperature event could cause the sensors to become mis-calibrated but that can be corrected by performing a recalibration with software such as the Dynamic Calibration tool or the RealSense Viewer's on-chip calibration feature.