Auto exposure when the camera is in inter cam sync mode 259 / 260

[jeferal]

Required Info
Camera Model	D455
Firmware Version	5.16.0.1
Operating System & Version	Linux (Ubuntu 20)
Kernel Version (Linux Only)	5.15.49
Platform	PC
SDK Version	2.55.1
Language	C++
Segment	Robot

Issue Description

Hello, I am interested in using the Realsense cameras in a gen locked mode and taking infrared frames with emitter On and emitter Off, but I am having trouble at having an auto exposure. It seems that when the camera is working in inter cam sync mode 260, the exposure is fixed even if the auto exposure option is enabled.

After seeing this issue: https://github.com/IntelRealSense/librealsense/issues/11908#issuecomment-2013161828 I would like to confirm if it is possible that auto exposure does not work when inter cam sync mode is set to 260 the same way it does not work when the option emitter on off is enabled (both gain level and exposure time do not seem to change when using SDK 2.55.1 and FW 5.16.0.1).

I have also noticed differences in emitter on off and auto exposure behaviours between different SDK and FW versions. In these two videos, I am trying to compare one camera using realsense viewer with SDK version 2.50.0 and camera with FW version 5.13.0.50 with emitter on off enabled and it is possible to see in the metadata that although the exposure time does not vary, the gain does vary (this would also be suitable for me).

https://github.com/IntelRealSense/librealsense/assets/62122095/04eac1da-f87d-4013-b045-6edf6a2844fd

However, when using SDK version 2.55.1 and a camera with FW 5.16.0.1 (emitter on off enabled) the metadata shows that both the exposure time or the gain do not vary. What is the reason of this?

https://github.com/IntelRealSense/librealsense/assets/62122095/4ba536a7-72c1-468a-ae4e-30d17e0b55d4

Thank you very much for your help in advance!

[MartyG-RealSense]

Hi @jeferal The main change to exposure and gain between SDK versions 2.50.0 and 2.55.1 is that in 2.55.1 and firmware 5.16.0.1, the Auto-Exposure Limit and Auto-Gain Limit options were re-introduced for all 400 Series camera models except D415 and D457. These options - originally introduced in SDK 2.42.0 - are described below.

RS2_OPTION_AUTO_EXPOSURE_LIMIT Limit maximal exposure used in Auto-Exposure to preserve FPS.

RS2_OPTION_AUTO_GAIN_LIMIT Limit maximal digital gain value used in Auto-Exposure.

Regarding auto-exposure not working in Inter Cam Sync Mode 259 and 260: I would think that it would almost certainly have the same auto-exposure characteristic as https://github.com/IntelRealSense/librealsense/issues/11908#issuecomment-2013161828 because the IR emitter is being turned on and off by these modes.

However, it is not something that can be officially tested for confirmation as Intel no longer provide technical support for the genlock hardware sync modes, only for the original Inter Cam Sync Modes 1 and 2 (though the genlock modes continue to be usable in the SDK).

[jeferal]

Hi @MartyG-RealSense, thank you very much for your quick reply and your detailed explanation, I really appreciate it. I am also aware of the new release of FW and SDK, which supports these limit parameters and I understand that the modes 259 and 260 are not supported.

However, how would you explain that with the new FW 5.16.0.1 and SDK 2.55.1 when having the stereo sensor in emitter on off mode and auto exposure, the gain does not vary. But when using the FW 5.13.0.50 and SDK 2.50.0 the gain varies when there are changes in the intensity of the environment?

[MartyG-RealSense]

There have been so many changes to the SDK since 2.50.0 that it would be very difficult to identify precisely what could have caused an apparent change in exposure / gain behaviour when using genlock hardware sync, especially as the effect of changes on genlock modes would likely not be tested for as new SDK versions are released, since genlock is no longer supported. I do apologize.

[jeferal]

Thank you so much. I completely understand. The differences that I pointed in the videos were happening when the camera is not in gen locked mode. It is a configuration of a single camera, working with the sensor option emitter_on_off set to true. This is the scenario where I notice the difference in how autoexposure works among different firmware/sdk versions. I apologize for not being very clear.

[MartyG-RealSense]

I checked the firmware release notes. Firmware 5.15.0.2 (which corresponds to SDK 2.54.1) added an enhanced auto-exposure algorithm for D455.

[jeferal]

Thank you @MartyG-RealSense, I will bear that in mind. One more question, is it possible that you could confirm me that auto exposure does not work in the latest FW version 5.16.0.1 and SDK version 2.55.1 when a single camera (D455) is working in emitter_on_off mode? When I mean that it does not work, I mean that both the exposure time and the gain are fixed.

Thank you very much for your help again.

[MartyG-RealSense]

I ran tests to observe the Actual Exposure metadata in RealSense Viewer 2.55.1 with a D455 and firmware 5.15.0.2 and then 5.16.0.1. Actual Exposure updated when auto-exposure was enabled and emitter on_off was disabled.

Then I tested with auto-exposure enabled and emitter_on_off enabled. With both firmware 5.15.0.2 and 5.16.0.1 the Actual Exposure did not update when emitter on_off was enabled and gain did not change.

THen I tested it in older Viewer versions with on-off enabled. In 2.54.2 and 2.54.1 the Actual Exposure and Gain did not change.

Finally, I tested with Viewer 2.50.0 and firmware 5.13.0.50 (as newer firmwares than that do not work with 2.50.0). Actual Exposure and Gain were able to change. This replicates what you experienced.

However, because 5.13.0.50 is the newest firmware that works with 2.50.0, it cannot be confirmed whether they are able to change because of SDK 2.50.0 or firmware 5.13.0.50.

[jeferal]

I really appreciate your effort in conducting these tests, the information is very valuable.

Therefore, what is the expected behaviour of auto exposure when it works together with emitter on_off?

Just to understand if the fact that both exposure time and gain level do not change is a bug or if it is supposed to work this way.

[MartyG-RealSense]

The earlier opinion of my Intel RealSense colleagues at https://github.com/IntelRealSense/librealsense/issues/11908#issuecomment-2013161828 was that when emitter_on_off is enabled, auto-exposure should be fixed, otherwise the auto-exposure would not have time to settle the exposure down when frequently alternating the emitter.

[jeferal]

Thank you very much @MartyG-RealSense. That makes sense. The application I am working with is quite extreme in the sense that I require a IR picture with emitter on and a IR picture with emitter off as well as being able to very different light conditions (that's why I was interested in the auto exposure). Is there any way I could achieve this with the D455 camera?

[MartyG-RealSense]

If you are using hardware sync then mode 259 / 260 is the only way to control emitter on-off over multiple cameras, otherwise each camera will be individually controlling its on-off unsychronized with the other cameras.

The High-Dynamic Range (HDR) feature enables the camera to handle very different lighting conditions by enabling two different exposure values to be used so that the camera can cope with scenes that have both light and dark areas. When using HDR though, auto-exposure is disabled and emitter on-off cannot be used.

https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

[MartyG-RealSense]

Hi @jeferal Do you require further assistance with this case, please? Thanks!