D435 Camera, The edge of object problem

[GHCK57]

---

Required Info
Camera Model	{ D400 }
Firmware Version	05.12.15.50)
Operating System & Version	{Win (10)
SDK Version	{ legacy / 2.<?>.<?> }
Language	{python }
Segment	{others }

Issue Description

Hi,

While working with Realsense D435 camera I faced with a problem. When the camera is close to object approximately 35-40 cm, depth frame looks good. But when I put the camera more farther approximately 70cm. The edges of object are deformed.

I use the "opencv_pointcloud_viewer.py " shared by Realsense to get the point cloud data. This code -->

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_pointcloud_viewer.py

That is what I want to measure. There is a surface and there is a object on this surface. I am just looking the edges.

This photo shows the .ply file taken from camera 70 cm far away

And also this one is taken from 35 cm far away

As you can see, there is a clear difference in the edges of the object. I have to work 70cm far away. Why does this difference occur? How can I deal with this problem. I just want to flat edge.

I hope you understood me. I wait for your helping. Thanks a lot.

[MartyG-RealSense]

The above image with the screwdrivers either side of the box reminds me of when I tested creating a pointcloud of thin toothpicks in https://github.com/IntelRealSense/librealsense/issues/8228 at a similar distance range. The toothpicks had the same kind of black area around them at first. I was able to remove the black areas by using a depth unit scale of '0.0001' like you tried in https://github.com/IntelRealSense/librealsense/issues/10133#issuecomment-1007155390

The speckled effect on all the surfaces in the above image makes me think that the IR dot pattern might be leaking through onto the RGB image, something that is not meant to happen. When it does happen, reducing the Laser Power value can reduce the visibility of the dots.

[gercekcetinkaya]

Hi @MartyG-RealSense It is @GHCK57

What is the meaning of leakage IR dot pattern onto the RGB image ? I will check the changing the Laser Power to visibility of the dots tomorrow. In your comment #8228, the depth measurement on of the toothpick is very successful. Why can't I get a healthy depth data?

** Also I saw your comment in https://support.intelrealsense.com/hc/en-us/community/posts/360035147633/comments/360009059553 here. My camera's temperature also is high, approximately 50 degree. I will try the solutions you posted in there. I will share the results with you.

[MartyG-RealSense]

An infrared image is covered in a projection of semi-random dots from the camera's IR Emitter component, like your IR images above. These dots may appear on the depth image as black dots, an undesirable noise effect called laser speckle. They should not be appearing on the RGB image though, and when they do it can be described as a 'leaking through'. I am not aware of negative effects to the RGB image of such a leak though, other than it looking wrong in terms of the real-life scene not having dots on it.

In regard to the toothpick image: a characteristic of the scene that the camera was in is that it does not have strong lighting. I also put a plain flat background card behind the toothpick and set the depth scale to 0.0001. The depth scale change by far made the largest impact on removing unwanted black and grey areas from the image.

I also experimented with setting the Disparity Shift option (under Advanced Mode > Depth Table in the Viewer) to a value of '50' instead of the default '0' to reduce the camera's minimum depth sensing distance below the default 0.1 meters of D435 / D435i.

[GHCK57]

I tested the changing of temperature. It take approximately 30 minutes to reach the 44 degree temperature. While testing just color and depth frame was open and Laser power was 150mW. There was no error occurred.

The second test ,s changing the Laser Power.

The laser Power is 0:

The laser power is 50:

The laser Power is 150:

The laser Power is 300:

[MartyG-RealSense]

It can be observed that as Laser Power increases, the number of black dots increases, as shown in your '300' image. In those particular images, the speckle effect seems to be mostly in the area away from the box rather than negatively affecting the box itself even though the dot pattern is being cast onto the box.

This is an expected effect, as the dot visibility can differ depending on the type of material that a surface is made from and the time of day. It has been demonstrated that earlier in the day the pattern may be invisible on one type of material in the scene and strongly visible on another type of material, whilst in late afternoon as light levels reduce, the visibility pattern may reverse, with the formerly invisible area of dots now visible and the strongly visible dots becoming invisible.

[GHCK57]

Then, What sholuld I do ? What is your suggestions ? There is huge black hole apperings behind of the box. When the lights turn off it does not appear. Also there are lots of problem like in https://github.com/IntelRealSense/librealsense/issues/10133#issuecomment-1015261451 . The camera does not see very well the box, how can I measure of its volume ?

[MartyG-RealSense]

I recommend determining the lighting level and settings in which the dots are most visible on the box's surface. In your particular scene, that would seem to be Laser Power = 300 and with the lights turned off. The camera can then use the dots on the surface of the box as a 'texture source' to analyze for depth detail. This approach works well for plain surfaces with a low amount of texture detail on them like the cardboard.

As mentioned earlier in https://github.com/IntelRealSense/librealsense/issues/10133#issuecomment-1007310195 the RealSense Viewer allows manual selection of multiple points on a pointcloud to calculate the area.

The link below suggests an approach for using PCL to calculate volume.

https://stackoverflow.com/questions/55629892/2019-point-cloud-volume-estimation-with-c/55681149#55681149

Another approach would be to use a RealSense-powered commercial box measurement solution such as the ones in the links below.

https://www.mobileworxs.com/volume-dimensioning-3d-camera-system/ https://www.lips-hci.com/lipsmetric

[GHCK57]

In summary, the camera can be affected by the light source, even if the light source is soft. It is necessary to store the camera so that it is not affected by external sources.

But despite everything, I couldn't get the performance I wanted from the camera. I still haven't found the solutions to the questions I wrote earlier. For example, as in comment (https://github.com/IntelRealSense/librealsense/issues/10133#issuecomment-1014584236) and https://github.com/IntelRealSense/librealsense/issues/10133#issuecomment-1015175325, I could not find why the edges of the box were trying to connect to the table surface.

[MartyG-RealSense]

I cannot think of a way to deal with the table surface other than to position the camera at a top-down angle at a fixed height and set the threshold filter post-processing filter with a Maximum distance value where depth data at the table height and below is excluded from the depth image. Even if larger boxes are used, if the table top is a fixed distance from the camera then the table under the base of the box should always be removed whilst the box remains on the image.

[GHCK57]

For this job. The D415 cameras is better than D435, isn't it ?

[MartyG-RealSense]

I did consistently obtain better box images with the D415 model in my own tests with a similar sized cardboard box. The D415, though it has a smaller field of view, has around 2x better image quality than D435 and 2x less RMS error (noise over distance). It also has an optimal depth accuracy resolution of 1280x720, compared to 848x480 on D435 / D435i. The D415 excels at scanning small static objects, whilst the D435 is optimized for capturing fast motion.

I cannot guarantee that a D415 will resolve the issues that you are experiencing in your own particular scene though. I can only speak for my own test results.

[GHCK57]

"The D415 excels at scanning small static objects, whilst the D435 is optimized for capturing fast motion." Is this diffrence thanks to shutter ?

->Is there a planned production end time for these cameras?

-->Does the D415 camera works with same code that written for D435 ?

[MartyG-RealSense]

1. The D415 has a smaller field of view (FOV) on its left and right infrared sensors compared to the D435, which has a wider IR sensor component. A wider sensor lets more light into it, which can result in greater depth noise.

2. There are no announced plans for EOL of the 400 Series at the time of writing this. My understanding is that the 400 Series will be supported at least through the whole of 2022.

3. The D415 works with the same code as the D435. An exception to this would be code that looks for a particular camera model name (which would prevent a D435 from working with the script if it only accepted a D415), but most scripts would not use such a check.

[GHCK57]

I am very pleased. Thank you very much for your help and information. I will buy D415 camera then I will try with this.

[MartyG-RealSense]

Hi @GHCK57 Do you require further assistance with this case, please? Thanks!

[GHCK57]

Hi @MartyG-RealSense thanks for your help.

[MartyG-RealSense]

Thanks very much @GHCK57 for the update. As you are satisifed with the outcome, I will close this case. Thanks again!