Overexposed IR images in sunlight with D455

DanMesh commented 2 years ago

The problem

When using a D455 camera outdoors in sunlight, the infrared images are often very overexposed. When this happens, the depth image contains no data in the overexposed regions. I am looking for any insights as to what might be causing this issue, or suggestions to try and mitigate this problem.

The following images show examples of the depth (top) and IR (bottom) images on two of these occasions. The camera is facing a tree in a grassy field. The leaves and grass are very washed-out, and (I assume) the lack of any texture means the depth cannot be inferred. The config files and resulting syslogs are below.

depth_and_image_1 depth_and_image_2

Investigating auto-exposure

Since this looks like an exposure issue, I tried to play around with the auto-exposure settings. I reduced the auto-exposure limit as low as 120 (microseconds), but that had almost no effect on the resulting IR image. Reducing the limit further drastically degraded the image quality. This leads me to believe the auto-exposure is already lowering the exposure setpoint as much as it can, but that is not enough.

Appendices

Config: `d455.yaml`

camera:
    camera:
      ros__parameters:
        device_type: 'd455'
        serial_no: ''
        usb_port_id: ''

        json_file_path: "/home/dan/dev/realsense_ws/src/realsense-ros/realsense2_camera/config/camera-depth-filter-settings.json"

        depth_fps: 10.0
        depth_width: 848
        depth_height: 480
        depth_qos: "SENSOR_DATA"

        infra_fps: 10.0
        infra_width: 848
        infra_height: 480
        infra_qos: "SENSOR_DATA"

        enable_accel: false
        enable_color: false
        enable_depth: true
        enable_gyro: false
        enable_infra1: true
        enable_infra2: false
        enable_pointcloud: false

        filters: ''

        pointcloud_texture_index: 0
        pointcloud_texture_stream: RS2_STREAM_COLOR

        stereo_module:
          emitter_enabled: 0
        unite_imu_method: ''

        rosbag_filename: ''

Config: `camera-depth-filter-settings.json`

{
  "aux-param-autoexposure-setpoint": "2000",
  "aux-param-colorcorrection1": "0.298828",
  "aux-param-colorcorrection10": "0",
  "aux-param-colorcorrection11": "0",
  "aux-param-colorcorrection12": "0",
  "aux-param-colorcorrection2": "0.293945",
  "aux-param-colorcorrection3": "0.293945",
  "aux-param-colorcorrection4": "0.114258",
  "aux-param-colorcorrection5": "0",
  "aux-param-colorcorrection6": "0",
  "aux-param-colorcorrection7": "0",
  "aux-param-colorcorrection8": "0",
  "aux-param-colorcorrection9": "0",
  "aux-param-depthclampmax": "65536",
  "aux-param-depthclampmin": "0",
  "aux-param-disparityshift": "0",
  "controls-autoexposure-auto": "True",
  "controls-autoexposure-manual": "8500",
  "controls-depth-gain": "16",
  "controls-laserpower": "0",
  "controls-laserstate": "on",
  "ignoreSAD": "0",
  "param-autoexposure-setpoint": "2000",
  "param-censusenablereg-udiameter": "9",
  "param-censusenablereg-vdiameter": "9",
  "param-censususize": "9",
  "param-censusvsize": "9",
  "param-depthclampmax": "65536",
  "param-depthclampmin": "0",
  "param-depthunits": "1000",
  "param-disableraucolor": "0",
  "param-disablesadcolor": "0",
  "param-disablesadnormalize": "0",
  "param-disablesloleftcolor": "0",
  "param-disableslorightcolor": "1",
  "param-disparitymode": "0",
  "param-disparityshift": "0",
  "param-lambdaad": "751",
  "param-lambdacensus": "6",
  "param-leftrightthreshold": "10",
  "param-maxscorethreshb": "1423",
  "param-medianthreshold": "625",
  "param-minscorethresha": "4",
  "param-neighborthresh": "108",
  "param-raumine": "6",
  "param-rauminn": "3",
  "param-rauminnssum": "7",
  "param-raumins": "2",
  "param-rauminw": "2",
  "param-rauminwesum": "12",
  "param-regioncolorthresholdb": "0.784736",
  "param-regioncolorthresholdg": "0.565558",
  "param-regioncolorthresholdr": "0.985323",
  "param-regionshrinku": "3",
  "param-regionshrinkv": "0",
  "param-robbinsmonrodecrement": "5",
  "param-robbinsmonroincrement": "5",
  "param-rsmdiffthreshold": "1.65625",
  "param-rsmrauslodiffthreshold": "0.71875",
  "param-rsmremovethreshold": "0.809524",
  "param-scanlineedgetaub": "13",
  "param-scanlineedgetaug": "15",
  "param-scanlineedgetaur": "30",
  "param-scanlinep1": "30",
  "param-scanlinep1onediscon": "76",
  "param-scanlinep1twodiscon": "86",
  "param-scanlinep2": "98",
  "param-scanlinep2onediscon": "105",
  "param-scanlinep2twodiscon": "33",
  "param-secondpeakdelta": "775",
  "param-texturecountthresh": "4",
  "param-texturedifferencethresh": "50",
  "param-usersm": "1",
  "param-zunits": "1000"
}

Syslogs

[INFO] [launch]: All log files can be found below /home/dan/.ros/log/2022-04-14-14-17-00-065027-padfoot-248958
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [realsense2_camera_node-1]: process started with pid [248960]
[realsense2_camera_node-1] [INFO] [1649938620.156543506] [RealSenseCameraNode]: RealSense ROS v3.2.1
[realsense2_camera_node-1] [INFO] [1649938620.156582583] [RealSenseCameraNode]: Built with LibRealSense v2.45.0
[realsense2_camera_node-1] [INFO] [1649938620.156589807] [RealSenseCameraNode]: Running with LibRealSense v2.45.0
[realsense2_camera_node-1] [INFO] [1649938620.166679376] [RealSenseCameraNode]: Device with serial number 031422250089 was found.
[realsense2_camera_node-1] 
[realsense2_camera_node-1] [INFO] [1649938620.166693403] [RealSenseCameraNode]: Device with physical ID /sys/devices/pci0000:00/0000:00:14.0/usb3/3-7/3-7:1.0/video4linux/video2 was found.
[realsense2_camera_node-1] [INFO] [1649938620.166697027] [RealSenseCameraNode]: Device with name Intel RealSense D455 was found.
[realsense2_camera_node-1] [INFO] [1649938620.166835493] [RealSenseCameraNode]: Device with port number 3-7 was found.
[realsense2_camera_node-1] [INFO] [1649938620.166842306] [RealSenseCameraNode]: Device USB type: 2.1
[realsense2_camera_node-1] [WARN] [1649938620.166845543] [RealSenseCameraNode]: Device 031422250089 is connected using a 2.1 port. Reduced performance is expected.
[realsense2_camera_node-1] [INFO] [1649938620.167452313] [RealSenseCameraNode]: getParameters...
[realsense2_camera_node-1] [INFO] [1649938620.168057105] [RealSenseCameraNode]: setupDevice...
[realsense2_camera_node-1] [INFO] [1649938620.503789774] [RealSenseCameraNode]: JSON file is loaded! (/home/dan/dev/realsense_ws/src/realsense-ros/realsense2_camera/config/camera-depth-filter-settings.json)
[realsense2_camera_node-1] [INFO] [1649938620.503812810] [RealSenseCameraNode]: Device Name: Intel RealSense D455
[realsense2_camera_node-1] [INFO] [1649938620.503816551] [RealSenseCameraNode]: Device Serial No: 031422250089
[realsense2_camera_node-1] [INFO] [1649938620.503820887] [RealSenseCameraNode]: Device physical port: /sys/devices/pci0000:00/0000:00:14.0/usb3/3-7/3-7:1.0/video4linux/video2
[realsense2_camera_node-1] [INFO] [1649938620.503823910] [RealSenseCameraNode]: Device FW version: 05.12.15.50
[realsense2_camera_node-1] [INFO] [1649938620.503826600] [RealSenseCameraNode]: Device Product ID: 0x0B5C
[realsense2_camera_node-1] [INFO] [1649938620.503829287] [RealSenseCameraNode]: Enable PointCloud: Off
[realsense2_camera_node-1] [INFO] [1649938620.503831840] [RealSenseCameraNode]: Align Depth: Off
[realsense2_camera_node-1] [INFO] [1649938620.503834388] [RealSenseCameraNode]: Sync Mode: Off
[realsense2_camera_node-1] [INFO] [1649938620.503841431] [RealSenseCameraNode]: Device Sensors: 
[realsense2_camera_node-1] [INFO] [1649938620.504827378] [RealSenseCameraNode]: Stereo Module was found.
[realsense2_camera_node-1] [INFO] [1649938620.505613152] [RealSenseCameraNode]: RGB Camera was found.
[realsense2_camera_node-1] [INFO] [1649938620.505686213] [RealSenseCameraNode]: Motion Module was found.
[realsense2_camera_node-1] [INFO] [1649938620.505698500] [RealSenseCameraNode]: (Fisheye, 0) sensor isn't supported by current device! -- Skipping...
[realsense2_camera_node-1] [INFO] [1649938620.505704873] [RealSenseCameraNode]: (Fisheye, 1) sensor isn't supported by current device! -- Skipping...
[realsense2_camera_node-1] [INFO] [1649938620.505709657] [RealSenseCameraNode]: (Fisheye, 2) sensor isn't supported by current device! -- Skipping...
[realsense2_camera_node-1] [INFO] [1649938620.505715221] [RealSenseCameraNode]: (Pose, 0) sensor isn't supported by current device! -- Skipping...
[realsense2_camera_node-1] [INFO] [1649938620.505720781] [RealSenseCameraNode]: (Confidence, 0) sensor isn't supported by current device! -- Skipping...
[realsense2_camera_node-1] [INFO] [1649938620.505727532] [RealSenseCameraNode]: num_filters: 0
[realsense2_camera_node-1] [INFO] [1649938620.505731140] [RealSenseCameraNode]: Setting Dynamic reconfig parameters.
[realsense2_camera_node-1] set_option: stereo_module.emitter_enabled = 0
[realsense2_camera_node-1] [WARN] [1649938620.575476826] [RealSenseCameraNode]: Could not set param: rgb_camera.power_line_frequency with 3Range: [0, 2]: parameter 'rgb_camera.power_line_frequency' could not be set: Parameter {} doesn't comply with integer range.
[realsense2_camera_node-1] [INFO] [1649938620.575600001] [RealSenseCameraNode]: Done Setting Dynamic reconfig parameters.
[realsense2_camera_node-1] [INFO] [1649938620.576027271] [RealSenseCameraNode]: depth stream is enabled - width: 848, height: 480, fps: 10, Format: Z16
[realsense2_camera_node-1] [WARN] [1649938620.576169051] [RealSenseCameraNode]: Given stream configuration is not supported by the device!  Stream: Infrared, Stream Index: 0, Width: 848, Height: 480, FPS: 10, Format: Y8
[realsense2_camera_node-1] [INFO] [1649938620.576400429] [RealSenseCameraNode]: infra1 stream is enabled - width: 848, height: 480, fps: 10, Format: Y8
[realsense2_camera_node-1] [INFO] [1649938620.576409940] [RealSenseCameraNode]: setupPublishers...
[realsense2_camera_node-1] [INFO] [1649938620.577854918] [RealSenseCameraNode]: setupStreams...
[realsense2_camera_node-1] [INFO] [1649938620.735342353] [RealSenseCameraNode]: SELECTED BASE:Depth, 0
[realsense2_camera_node-1] [INFO] [1649938620.735592469] [RealSenseCameraNode]: RealSense Node Is Up!
[realsense2_camera_node-1] [WARN] [1649938620.909186039] [RealSenseCameraNode]:

MartyG-RealSense commented 2 years ago

Hi @DanMesh When using RealSense 400 Series cameras in strong sunlight, if the sun is visible in the camera's field of view, Intel recommend in their camera tuning guide to set an auto-exposure Region Of Interest (ROI) in the lower half of the image.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#section-use-sunlight-but-avoid-glare

https://github.com/IntelRealSense/librealsense/issues/2190#issuecomment-589027974 offers advice for defining an auto-exposure ROI in the RealSense ROS wrapper using rosparam instructions in the launch file.

DanMesh commented 2 years ago

Thanks for the feedback @MartyG-RealSense. I have tested my setup with the ROI in the lower half of the FOV, but this has had little-to-no effect. From my experiments, it seems that the auto-exposure is already setting the exposure to the lowest value it can (regardless of ROI), but this is still not low enough.

The scenario in which I am observing this is when the sun is behind the camera, and the reflections/glare are very strong. If a very thin layer of cloud covers the sun, it already makes a big difference in improving the IR and depth images. This makes me think a physical filter on the camera might be helpful for reducing the incoming light.

MartyG-RealSense commented 2 years ago

A physical filter is likely to make a positive difference. You could also try setting the camera's IR emitter to rapidly alternate on and off to see whether that makes a positive difference. A RealSense team member suggests instructions at https://github.com/IntelRealSense/realsense-ros/issues/1657#issuecomment-968920694 for doing this with the RealSense ROS2 3.x wrapper (as you are using 3.2.1).

DanMesh commented 2 years ago

Thanks, I will see if I can try a physical filter.

I tried enabling the emitter_on_off setting, but that produces the following error:

Failed to set value: hwmon command 0x7b( 17 0 0 0 ) failed (response -6= Invalid parameter)

This also happens if I try to enable the setting in the RealSense Viewer.

Adding an initial reset (as you suggested in https://github.com/IntelRealSense/librealsense/issues/8994#issuecomment-841295372) did not solve the problem.
Elsewhere, I have seen comments saying that this was an issue with firmware version 5.11 and that reverting to 5.10 could solve the problem. I am on version 05.12.15.50, so I'm not sure if this would be an appropriate solution.
Similarly, in https://github.com/IntelRealSense/librealsense/issues/8004 you suggested retrying to set the parameter multiple times, but this has also not worked.

MartyG-RealSense commented 2 years ago

The minimum firmware recommended for D455 is 5.12.7.100 so using an earlier firmware than that could cause harm to your camera.

It seems from your syslog that you are using Linux as your OS. Is that correct, please?

DanMesh commented 2 years ago

Yes, that's correct. Those tests were done on an Ubuntu 20.04.4 system with ROS2 Foxy.

MartyG-RealSense commented 2 years ago

The infrared image that you are experiencing is similar to one that a RealSense user had outdoors in 2018 at https://github.com/IntelRealSense/librealsense/issues/2875

A RealSense team member in this case suggests defining a ROI (which you have already tried) or using a filter (which you are planning to try).

MartyG-RealSense commented 2 years ago

Hi @DanMesh Do you require further assistance with this case, please? Thanks!

DanMesh commented 2 years ago

@MartyG-RealSense For now I will continue investigating myself, so we can close this issue. Thanks for your assistance!

MartyG-RealSense commented 2 years ago

You are very welcome, @DanMesh - thanks very much for the update! As you suggested, I will close the issue. Thanks again!

IntelRealSense / realsense-ros