About the noise problem after D435 is deep converted to laser

[ju-mingyue]

Required Info
Camera Model	D435
Firmware Version	05.12.13.50
Operating System & Version	Linux (Ubuntu18.04.5)
Kernel Version (Linux Only)	4.9.201
Platform	NVIDIA JetsonNano B01
SDK Version	2.48.0
Language	ROS packages
Segment	Robot

Issue Description

Hello, when I use the D435 camera, I use the depthimage_to_laserscan package to convert the depth information into laser information, but the problem also arises. Although it is possible to add an obstacle in the costmap, it will suddenly add an obstacle where there is no obstacle. , I think this is caused by the presence of noise in the camera. So I hope to help me solve this problem. Or give me some useful suggestions.

The yellow part of the picture above is the barrier-free area, but the robot will add obstacles in this part, and there is glass in these areas. I think this has a certain impact on the camera? How do I need to configure it to avoid the influence of glass on it.

[MartyG-RealSense]

Hi @ju-mingyue If the scene that the camera is in is well lit then you can reduce noise by disabling the camera's IR emitter to remove the semi-random dot pattern that it casts on the scene (which can create a noise phenomenon called laser speckle). The camera can then alternatively use the ambient light in the scene to analyze surfaces for depth information. If the scene is not well lit then disabling the IR emitter would result in reduced depth-image detail.

The negative effect of glare from reflective surfaces such as glass can be greatly reduced by applying a physical optical filter product called a linear polarization filter over the lenses on the outside of the camera. More information about this can be found in section 4.4 When to use polarizers and waveplates of Intel's white-paper document about optical filters.

https://dev.intelrealsense.com/docs/optical-filters-for-intel-realsense-depth-cameras-d400#section-4-the-use-of-optical-filters

An example of the difference that a linear polarization filter makes to capturing glass is shown in the image from the paper below.

[ju-mingyue]

@MartyG-RealSense Thank you for your prompt reply. My scene is a very well-lit environment, so I can try to turn off the infrared transmitter you mentioned, but I would like to ask, how to turn off the infrared transmitter of the camera? Hope you can guide me further.

[MartyG-RealSense]

In ROS you could try disabling it with a rosparam in the launch file (see the link below) or by using emitter_enabled:=false in the roslaunch instruction.

https://github.com/IntelRealSense/realsense-ros/issues/1379#issuecomment-691453842

In the above link, the RealSense ROS user in that case chose to instead disable the emitter in the base node itself by default.

[ju-mingyue]

I added it in the launch file, but the following problem appeared, is it because my D435 does not support closing?

RLException: unused args [emitter_enable] for include of [/home/hanning/robot_ws/src/realsense-ros-2.3.1/realsense2_camera/launch/includes/nodelet.launch.xml] The traceback for the exception was written to the log file

[MartyG-RealSense]

The D435 is able to have its emitter disabled. I have not seen that particular error with RealSense ROS before, but the link below suggests that it is a general ROS error related to how the instructions in the launch file have been written.

https://answers.ros.org/question/239384/unused-args-for-include-of/

[ju-mingyue]

Thank you very much, I have added the command, but I did not see any obvious difference. How do I need to test to tell?

[MartyG-RealSense]

You could check the on / off status of the emitter during runtime in the dynamic_reconfigure interface by inputting the command below into the ROS terminal:

rosrun rqt_reconfigure rqt_reconfigure

https://github.com/IntelRealSense/realsense-ros#set-camera-controls-using-dynamic-reconfigure-params

[ju-mingyue]

Hello, I only found these two, but can I write this command directly into the launch file? Or choose other methods?

[MartyG-RealSense]

The emitter_enabled control has now been moved further up the interface and put under a drop-down menu instead of a tick-box. The state shown in your image - Laser (1) - means that the emitter is enabled. Please open the drop-down menu and see if you can set it to Off (the '0' state of the emitter) to disable it.

The methods mentioned in https://github.com/IntelRealSense/librealsense/issues/9556#issuecomment-891641986 are the only ones that I personally know of to change the emitter status in ROS1 during launch.

The emitter status should also be able to be changed during runtime with a dynamic_reconfigure text command without having to open the visual interface.

https://github.com/IntelRealSense/realsense-ros/issues/1379#issue-699601737

https://github.com/IntelRealSense/realsense-ros/issues/1206#issuecomment-632559220

A note about the above link https://github.com/IntelRealSense/realsense-ros/issues/1206#issuecomment-632559220 - you would use set stereo_module emitter_enabled 0 to disable the emitter instead of the '1' in the link, which would enable it.

[ju-mingyue]

@MartyG-RealSense Thank you very much for your prompt reply. Now this problem has mostly been solved, but there will still be non-existent obstacles near the glass, and after the depth information of our camera is converted into a laser signal, the position and laser seen in rviz The radar signal position is not uniform, and obstacles will be added very close.

My D435 is now installed at an angle of 45 degrees upwards

And at the same time, no matter how I configure it, I need to configure it again after each restart to shut down. I didn't understand how to make it shut down after restarting. The following is my configuration file, but emitter_enabled does not work, every time I need to adjust it in rqt.

`

/camera2/stereo_module/emitter_enabled: 0 /camera3/stereo_module/emitter_enabled: 0 /camera2/stereo_module/emitter_enabled: 1 /camera3/stereo_module/emitter_enabled: 1 ` The warning that appears after running is as follows: [ WARN] [1627961005.296478498]: Param '/d435_L/stereo_module/auto_exposure_limit' has value 0 that is not in range [1, 165000]. Removing this parameter from dynamic reconfigure options. [ WARN] [1627961005.302369956]: Param '/d435_L/stereo_module/auto_gain_limit' has value 0 that is not in range [16, 248]. Removing this parameter from dynamic reconfigure options. [ WARN] [1627961005.325473550]: Param '/d435_R/stereo_module/auto_exposure_limit' has value 0 that is not in range [1, 165000]. Removing this parameter from dynamic reconfigure options. [ WARN] [1627961005.334500946]: Param '/d435_R/stereo_module/auto_gain_limit' has value 0 that is not in range [16, 248]. Removing this parameter from dynamic reconfigure options. Use rqt to view, the results are as follows： 

[MartyG-RealSense]

The warnings in the launch log about removal of parameters from dynamic reconfigure options occur often in launch logs, so it should not be something to be concerned about.

If there are areas of depth information registering in places where there are not actually any objects then this may be due to a phenomenon called repetitive patterns. This is where the camera can become confused by horizontal or vertical rows of simlilar looking objects, such as a row of vertical fence posts or a stack of horizontal window blinds. This confusion can manifest on the depth image as floating blobs. Tilting the camera can help to break up the repetitive patterns and reduce the negative effect of the observed pattern of objects. The phenomenon is described in the link below.

https://github.com/IntelRealSense/librealsense/issues/6713

Would it be possible to provide an RGB image of the real-world scene where the blobs are occurring please, as this will help to identify potential repetitive patterns that are present in the scene.

[ju-mingyue]

@MartyG-RealSense Hello, the picture below is an image of my real world, and the position marked in yellow on the map is an obstacle that actually does not exist. I call it a "ghost obstacle".

For such an environment, can I solve it by adjusting the camera parameters? Or how can I set the camera to detect only a fixed height, for example, the height of more than two meters is ignored and not detected, and the distance of more than two meters is not detected by me. The following is the configuration of my depth information to laser, can I modify the parameters inside to solve some of the problems.

My D435 is tilted 45 degrees upwards, and I want to detect a 2 meter high obstacle, then how do I need to set the range_max parameter in depthimage_to_laserscan

[MartyG-RealSense]

The tiled ceiling shown in the color image could be identified as a repetitive pattern by the camera and cause it confusion.

You could try changing the clip_distance in the launch file listed above from -2 to 2 to set a maximum depth distance of 2 meters. The RealSense ROS wrapper documentation states that when the clip distance value is negative (minus value) then the limitation on distance is disabled.

Fluorescent lights such as ceiling strip lights can also cause noise on the image, because the hot gas inside them flickers at a rate that is difficult to see with the human eye. The disruption caused by such lights can be reduced if you use a camera FPS speed that is close to the operating frequency in Hz of the lights. For example, if the lights operate at 30 Hz (their 'power line frequency') then use 30 FPS, and if they operate at 60 Hz then use 60 FPS.

Fluorescent lights in European regions can be more of a problem for RealSense cameras, as such lights may operate at 50 Hz, and so there is not a closely matching RealSense FPS frequency because 50 FPS is not supported as a selectable speed by the cameras.

[ju-mingyue]

Thank you very much for your suggestion, it worked a lot for me.

In your opinion ,which situation/behavior will cause noise?

In my region, our light operate at 50 Hz, I want to know how to deal with this problem? @MartyG-RealSense

[MartyG-RealSense]

My understanding is that if you cannot set an FPS of 50 in a region that uses 50hz fluorescent lights then you can disable auto-exposure and set a manual exposure value instead. Although RealSense camera shutters have a fixed speed, you can simulate a shutter speed change by configuring a manual exposure-time value instead (since shutter speed is equivalent to exposure time).

Alternatively, the RealSense SDK provides a means to set the power line frequency to Off, 50, 60 or Auto as an anti-flickering control. If you need to set it in RealSense ROS, there is the /rgb_camera/power_line_frequency param.

In the RealSense Viewer, it can be set in the RGB section of the Viewer's options side-panel, in the Power Line Frequency option's drop-down menu.

It can also be set with a scripting instruction called RS2_OPTION_POWER_LINE_FREQUENCY.

The link below provides a C++ example of defining the setting.

https://github.com/IntelRealSense/librealsense/issues/7766#issuecomment-726932329

[ju-mingyue]

Hello, with regard to the filter you mentioned, where can I buy a filter that matches D435? Can the entire mirror surface of D435 be completely covered? Hope you can give me a purchase connection

[MartyG-RealSense]

There are not filters especially for RealSense, they should be cut by yourself to the size needed. My understanding is that you only place squares of filter over the lenses of the sensors that need filtering, not the entire front of the camera. On the 400 Series cameras, if you cover the infrared sensor then you should cover both the left and right sensor though.

Regarding where to purchase them, a RealSense team member has said "Any thin film polarizer will do that is linear. The circular polarizers used most often in 3D glasses won’t work".

You can Google for the term thin film linear polarization filter sheet and that provides a list of suppliers in search results. For quick reference, putting this search term into Amazon also shows some examples of filter sheet products.

[ju-mingyue]

@MartyG-RealSense I'm very sorry to trouble you. I want to know how the D435 is usually installed on the robot, including the angle of inclination and the installation position. Is there any specific requirements? Does it need to be completely exposed? For example, I install the D435 inside the robot and make a round hole at the position of the camera. Is this okay? I hope you can give me some pictures of the installation of the camera on the actual robot, I will be very grateful for your help!

[MartyG-RealSense]

The most important factor is that the lenses are not obscured by the mounting bracket or by the position that the camera s mounted in. For example, a camera that is mounted on the underside of the robot may have its view obscured by part of the robot's frame or casing, or by a shelf in front of the camera.

An example of a mounting that has a custom enclosure around it but still has a clear field of view is the underwater camera housing for a RealSense camera shown in the link below.

https://github.com/IntelRealSense/realsense-ros/issues/1723#issuecomment-805760519

An example of fully integrated RealSense technology in the front of a robot, with a clear unobstructed view, is the mobile retail inventory robot Tally.

https://www.simberobotics.com/platform/tally/

Numerous robots use this front-mounted approach with RealSense cameras, or mounted on the underside tilted downwards towards the ground so that the robot frame / casing does not enter its view. Another example of this is Xiaomi's new CyberDog robot product.

https://www.theverge.com/2021/8/10/22618043/xiaomi-cyberdog-robot-dog-quadruped-specs-price

Basically though, your approach of installing the camera behind a round hole is totally fine as long as you cut the hole positions precisely (like the CyberDog's camera area in the image above) so that the edge of the holes does not overlap the field of view of the lenses and obscure the view.

[ju-mingyue]

@MartyG-RealSense

Thank you very much for your help, but I also hope to set the resolution of the camera by using a json file, as shown below: However, when I run the program, the camera resolution is displayed as shown below： So, I want to learn how to configure it correctly. Can this solve the problem of frame loss The above problem occurs when I use two D435s on jetsonNano. I mainly use the depth information of D435s to convert into laser signals. The rgb image is only used to display real-time images, so I want to adjust the resolution. To solve this problem, what other solutions do you have for this problem?

[MartyG-RealSense]

In the librealsense SDK at least, you cannot set resolution and FPS with a json file because those stream settings get overridden by the pipeline when it starts (the rest of the settings in the json should be applied as expected when ROS loads the json).

In ROS, the proper way to define a custom resolution and depth is with the roslaunch instruction or in the launch file. In RealSense ROS wrapper 2.2.22 and newer, three factors need to be provided for each stream type that is being used in order to set a custom stream configuration - width, height and FPS.

If these three factors are not provided then the RealSense ROS wrapper deems the custom configuration to be invalid and applies the default stream configuration of the particular RealSense camera model being used.

Below is an example custom roslaunch configuration with rs_camera.launch that configures depth and color streams at 640x480 and 30 FPS.

roslaunch realsense2_camera rs_camera.launch depth_width:=640 depth_height:=480 depth_fps:=30 color_width:=640 color_height:=480 color_fps:=30

You can change the resolution and FPS values in the instruction to whatever you want them to be so long as that particular resolution and FPS combination is supported for the particular RealSense camera model that you are using

---

In regard to the Incomplete video frame detected! warning, information about this message in ROS can be found at https://github.com/IntelRealSense/realsense-ros/issues/1899#issuecomment-849837579

[MartyG-RealSense]

Hi @ju-mingyue Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.