Exposure gradient in stereo images on D455

[DanMesh]

Required Info
Camera Model	D455
Firmware Version	05.12.15.50
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.14.0
Platform	PC
SDK Version	N/A
Language	N/A
Segment	N/A

Issue Description

When observing environments exposed to bright, direct sunlight, the lower half of the image is very overexposed. No depth data can be extracted from the overexposed regions.

It appears that the camera itself is treating different parts of the image differently - either due to mechanical/optical reasons or software reasons. Could this be the case?

What I have tried: (on two different D455 cameras)

1. Auto-exposure enabled, with...
   • the ROI set to the full image or the top/bottom halves, and
   • varying exposure limits.
2. Manual exposure with the exposure and gain at their lowest-allowed values (1 and 16 respectively).
3. HDR enabled and disabled.
4. Post-processing enabled and disabled.
5. Recording images with the RealSense Viewer and realsense-ros.
6. Using a custom config preset and the "Default" preset.
7. Placing a filter (sunglasses) in front of the lenses
   • This is the only approach that has worked.
   • Adding a filter is therefore a possible solution, but I would like to investigate other solutions first and rule out the possibility of some other error in my hardware/setup.

This issue is a follow-up to the one I raised in https://github.com/IntelRealSense/realsense-ros/issues/2318.

---

Extended illustration

The following images were taken within a few minutes of each other from roughly the same position. The exposure was on manual and set to the lowest-allowed value (1).

The first pair shows the infrared images with the camera the right way up (left) and upside down (right). The second pair shows the corresponding colour images with the camera the right way up (left) and upside down (right). I have rotated the upside down images to make the comparison easier.

One can see that the part of the image that corresponds with the bottom of the camera is always overexposed.

Infrared stream:

Colour stream:

[MartyG-RealSense]

Hi @DanMesh The pink-purple color on the RGB image is a known phenomenon occurring in bright light conditions that is specific to the D455 model and is not a bug. It is discussed on page 8 (the final page) of the Intel PDF document at the link below which provides advice for dealing with the issue.

https://www.intelrealsense.com/download/13629/

It recommends adding an Neural Density (ND) filter similar to what you have done successfully with sunglasses, or increasing the FPS rate in order to improve saturation.

[DanMesh]

Thanks for the quick response @MartyG-RealSense.

The purple hue was not my initial concern - I am more interested in the exposure in the infrared streams. In the "right way up" images, the RGB image is purple but the tiles are (mostly) not overexposed, while in the infrared image the tiles are completely overexposed.

Are the two necessarily linked?

[MartyG-RealSense]

It looks as though the position of the over-exposure is being swapped when the camera is rotated, with the ground over-exposed when the camera is in the normal orientation and the sky over-exposed when the camera is rotated upside down.

As the image is not upside down when the camera is flipped, are you vertically flipping the image in software, please? If so, a RealSense team member advises in https://github.com/IntelRealSense/librealsense/issues/6023#issuecomment-597615585 that if the camera is rotated and then the image is rotated to orient it back to a normal angle then the calibration and parameters of the camera need to be adjusted too. This may not be a significant problem if you are not making use of the depth image though.

Both the infrared and RGB sensors can become saturated in bright sunlight, so they have that factor in common. As they are separate sensors though, one sensor is not affecting the other. They are just apparently being independently affected by the same strong light source. It looks like a normal reaction to saturation from sunlight and I do not see a fault in your hardware or software setup and you have tried the two main types of solution that Intel recommend (a physical filter product or setting a ROI in the lower half of the image).

Furthermore, you could try adjusting the camera's auto-exposure setpoint value in combination with a ROI in the lower half of the image. When Autoexposure is turned on, it will average the intensity of all the pixels inside of the defined ROI and will try to maintain this value at the predefined setpoint. In the RealSense Viewer, this option can be found under Stereo Module > Advanced Mode > AE Control

[DanMesh]

Yes, that's right: the overexposure is related to the image/camera frame, not the scene. This is what I am trying to understand.

In the initial description I had rotated the "upside down" images when uploading them so that they could be more easily compared to the "right way up" ones. No rotation or transformation was done during the actual image capture. The figure below shows the images as they were captured. In my application the camera changes orientation frequently, so I don't intend to change the rotation-related parameters/calibration.

What is unusual is that the camera appears capable of correctly exposing the scene, but only at the top of the image frame.

If it seems that ND filters are the only known solution to this, then I will need to proceed with those.

[MartyG-RealSense]

Sunlight may reflect off the ground and cause problems in the image, especially when the ground surface is wet. When glare from reflections is the cause of the problem, using a linear polarization filter can significantly reduce the negative impact.

This subject is discussed in section 4.4 When to use polarizers and waveplates of Intel's white-paper guide about optical filters.

https://dev.intelrealsense.com/docs/optical-filters-for-intel-realsense-depth-cameras-d400#section-4-the-use-of-optical-filters

Below is a before and after image showing the effects on glare reduction of applying a linear polarization filter over the camera lenses.

Any polarizer that is linear should work (except for the circular ones in 3D glasses), so they can be purchased at an affordable price. You can research examples of these products by searching stores such as Amazon for the term thin film linear polarization filter sheet

[bastianhjaeger]

Thanks @MartyG-RealSense for your quick and detailed response. I work with @DanMesh on the topic.

We still might misunderstand each other on the topic, though. What @DanMesh pointed out in his last comment is not related to wet surfaces or a surface at all.

Lets step back and have a look to the picture again. On the left the ground, the flat surface is in the lower part of the image, but on the right the flat surface is in the upper part of the image. Thus, the overexposed part is both times the lower part of the image. The region of overexposure seems to always be the lower region of the captured frame. @DanMesh tested it with two different camera.

Do you have any knowledge on this behavior?

[MartyG-RealSense]

Hi @bastianhjaeger It is appropriate to consider a surface and its reflectivity as a possible cause when investigating this kind of effect on an infrared image, even if the surface is eventually found not to be responsible (as seems to be the case here). https://github.com/IntelRealSense/librealsense/issues/2875 highlights a similar infrared image from a concrete-like surface in bright sunlight conditions.

Can you confirm whether you have tried adjusting the auto-exposure setpoint value as suggested earlier in this discussion at https://github.com/IntelRealSense/librealsense/issues/10534#issuecomment-1134282099 please?

[DanMesh]

In general, the surface certainly contributes to the overexposure. However, in these tests I have made an effort to eliminate this as the source of the problem. The only difference between the "right way up" and "upside down" images is that the camera was rotated 180°. The camera position, its direction, and the scene were unchanged. This means the light that reaches the camera should be the same in the two test cases. This is why we suspect the issue is introduced somewhere between when the light hits the first lens and when the infrared image is published.

Can you confirm whether you have tried adjusting the auto-exposure setpoint value as suggested earlier in this discussion at https://github.com/IntelRealSense/librealsense/issues/10534#issuecomment-1134282099 please?

In previous tests I have tried changing the auto-exposure setpoint. However, in the images above, manual exposure was used with the the lowest-allowed exposure value.

[MartyG-RealSense]

If your project requires having auto-exposure disabled (which will prevent automatic adjustments to lighting conditions) then you could try having exposure set to 1 but increase the gain instead of having it minimized at 16 to see whether it compensates for the bright light.

I cannot find anything else to try that you have not already tried yourself.

[MartyG-RealSense]

Hi @DanMesh and @bastianhjaeger Do you have an update about this case that you can provide, please? Thanks!

[DanMesh]

@MartyG-RealSense At the moment there are no updates. However, we're in the process of setting up tests using ND filters and we'll see how those work. I can provide an update once that has been done.

[MartyG-RealSense]

Thanks very much @DanMesh - good luck!

[MartyG-RealSense]

Hi again @DanMesh and @bastianhjaeger A new RealSense D435f camera model with built-in filters on the left and right sensors has now been released. Information about it can be found at https://github.com/IntelRealSense/librealsense/issues/10560

[MartyG-RealSense]

Hi @DanMesh and @bastianhjaeger Do you have an update about this case that you can provice, please? Thanks!

[DanMesh]

@MartyG-RealSense Yes, we've been able to address this issue for now using ND filters.

Specifically, I've mounted ND filters with 20% transmission (OD=0.75) to the stereo sensors and spent some time testing this setup in our application. I'm happy with the results, which are illustrated in the example below:

Here, the floor tiles in the lower half of the image were overexposed and this led to missing depth data. Adding the filters corrected the exposure and made for a much-improved depth image.

I have other filters on hand (lower-transmission ND filters and long-pass IR filters as in the new D435f) and will test those out to compare to the 20% ND filter. I will post any interesting observations from those tests here, but otherwise the main problem is solved and I consider this issue closed for now.

The only doubt that still remains is about why only half of the image is overexposed, as in my original question. For now I must assume it is a feature of the mechanical/optical properties of the D455.

[MartyG-RealSense]

Thanks very much for the update! It's great to hear that an ND filter resolved the issue for you, and the difference to the image with the filter applied is excellent. Thanks again!