IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.59k stars 4.82k forks source link

Missing data chunks when capturing white objects (D435) #9992

Closed codeandstuf closed 2 years ago

codeandstuf commented 2 years ago

Required Info
Camera Model { D400 }
Firmware Version (5.13.0.50)
Operating System & Version {Win (10))
Platform PC
SDK Version { legacy / 2.50.0 }
Language {Python/ Pyrealsnese2}
Segment {Robot/Smartphone/VR/AR/others }

Issue Description

I am having trouble capturing depth information on white-colored objects. My specific problem is when trying to read depth data from cows that have ambient lighting on one side (the picture below should help) One side of the cow is shaded and the other side receives ambient light. The day this was taken was overcast, but it still seemed to throw the data accuracy off. I am using a USB 3.1, 30 fps, and experience the problem with both videos and pictures. My program uses auto-exposure and if possible I would like to continue to use auto-exposure because the lighting in the environment is constantly changing. Through random testing, I have found that the camera D435 seems to struggle to get readings off any white objects. Any help or suggestions on how to improve reading would be appreciated.

Thanks in advance!

image

MartyG-RealSense commented 2 years ago

Hi @codeandstuf In the areas of the cow where the ambient light is at its most intense, it blots out the skin texture of the cow and effectively makes it a low-texture or textureless surface that is difficult for the camera to read depth detail from (like a plain door, wall or table). In these circumstances, you can use the camera's infrared dot pattern projection to read depth detail, as the dots act as an analyzable texture source in place of the detail on the object / surface itself.

The visibility of the dots to the camera can be increased by maximizing the Laser Power setting to '360' (its default value is '150'). In the RealSense Viewer, you can maximize Laser Power by using the slider beside the option under the Stereo Module > Controls section of the Viewer's options side-panel.

Normally the camera would also have difficulty reading depth detail from dark grey / black surfaces. This is because these color shades absorb light. This is not a problem in this particular scene though, as projecting a strong light source onto the scene can bring out texture detail from such colors. So the strong ambient light is an advantage in this particular situation. And the infrared dot-pattern projection should enable depth detail to still be read from the black areas of the cow if the ambient light weakens.

Where there are black empty areas with no depth detail on the image, you could fill these in by enabling the Hole-Filling Filter post-processing filter under the Post-Processing Filters section of the Viewer side-panel.

If you have the budget to purchase a physical optical filter product to overlay over the lenses on the outside of the camera, a linear polarization filter can significantly negate glare from reflections. Section 4.4 When to use polarizers and waveplates of Intel's white-paper document about optical filters provides further information about this subject.

https://dev.intelrealsense.com/docs/optical-filters-for-intel-realsense-depth-cameras-d400#section-4-the-use-of-optical-filters

The image below demonstrates the glare negation that a linear polarization filter can provide.

image

codeandstuf commented 2 years ago

Thank you @MartyG-RealSense! I have tried turing the lasers power up, but I attempted that on site and it's very possible I did not do it correctly. I will give that another try next time I am data collecting. The use of a linear polarization might be my best bet. Cost is a limitation for this project (its a research project and the real world use would be for farmers who are trying to save every penny they can), but if the addition of a optical filter improves the image quality enough I belive it would be worth it. Do you have any recommendations on linear polarization filters? It might not be for a few months but I will update this post with the results of any solution I attempt to implement.

Lastly I want to say a huge thank you @MartyG-RealSense. I have read many of your previous posts and your advice has been extremely helpful to me in the past.

MartyG-RealSense commented 2 years ago

You are very welcome. :)

A RealSense team member advised about linear polarization filters in a past case that "any thin film polarizer will do that is linear. The circular polarizers used most often in 3D glasses won’t work".

You could see some budget-priced examples of linear polarization filters by searching Amazon for thin film linear polarization

Information about a 3D printing blueprint for a clip to hold filters to the outside of a D435 camera model is at https://github.com/IntelRealSense/librealsense/issues/7384#issuecomment-697046206