435 camera, I set the frame rate to 60, the depth of the recorded video looks unstable and keeps jumping around

[nevynwong]

It actually looks like it might be skipping a bit more than this graph. Will this affect the facial depth sequence information I capture?

[MartyG-RealSense]

Hi @nevynwong I ran extensive tests to see how depth measurement fluctuation could be reduced. Altering the Temporal post-processing filter so that its minimum was zero and its maximum was halfway stabilized fluctuation so that measurements remained fixed until the camera position is changed. I moved the mouse cursor over the image to read the coordinates.

Although the fluctuations of the image looked intense, the actual changes in the measured numeric depth values were small enough that the default settings may be fine. This may be different for your particular project, so it is worth checking the measured depth value in real-time yourself.

If your depth image was generated by a program that you have written yourself instead of using the Viewer, it is worth bearing in mind that the Viewer applies a set of post-processing filters by default when launched, whilst self-created scripts have no post-processing filters applied by default and they have to be deliberately programmed into the application.

[nevynwong]

@MartyG-RealSense Ok, this is very important to me. I need exactly the depth information when the face changes slightly, so it is crucial for me if this flicker has a large effect on the depth.

[MartyG-RealSense]

As mentioned, when the Temporal filter was set to '0' minimum and about the halfway point for the maximum ('46' in my test) then the depth value for a particular coordinate remained constant unless the camera was moved or the observed object (myself in this case) moved. So applying these settings in a Temporal filter may be worth testing in your own project to see if it can provide similar stability.

[nevynwong]

@MartyG-RealSense The post-processing settings in the viewer are not in the record's bag file, so shouldn't we also do our own post-processing operations? What is the general default value for time filtering when post-processing the recorded video?

[MartyG-RealSense]

Yes, post-processing is not saved in the bag. After a bag is loaded into the Viewer, post-processing can be applied to the bag data in real-time.

The default values for the Temporal Filter are listed in a table in the link below.

https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#temporal-filter

[nevynwong]

The face recorded by the D415 camera has some stepped points after reconstructing the 3D point cloud. I have encountered this problem before, but now I can't solve it according to the previous method.

In addition, there are some defects in the face of the SR300 reconstruction, such as the vertical lines on the left side of the face. Is there any solution to the problem of the SR300? Which of these two cameras is more recommended for capturing the depth of facial expressions?

[MartyG-RealSense]

Hi @nevynwong In the case of the D415 issue with the steps, I wonder if applying edge smoothing with the Spatial Filter post-processing filter may help with this.

https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#spatial-edge-preserving-filter

Given the age of the SR300's camera model and its technological limitations, I would recommend using a 400 Series camera such as your D415 to take advantage of its significantly greater capabilities and the advantage of its built-in Vision Processor D4 processing hardware that the SR300 does not have. You can certainly test the Spatial Filter with the SR300 too though to see whether it resolves the vertical-line issue of the SR300 image.

[nevynwong]

I still can't solve this problem, I want to capture facial expressions, need the depth of the face accuracy as well as the effectiveness of some good, as if the SR300 looks better. If 415 can solve this problem, I will choose d415 to capture the depth change of expressions.

The top is for 300. Below is the result of 415.

[MartyG-RealSense]

The results are typical of what I have seen in past cases of the difference between such 3D face reconstructions made in the SR300 and the RealSense 400 Series, with the SR300 model being more simple in its detail and texture. Aside from the age of the SR300, it uses a different camera technology (Structured Light) to the technology of the 400 Series (Stereo Depth). So there may be a limit to how closely you can get outputs from the two camera types to match.

[nevynwong]

My concern now is how to make the 415 result better, but I can't solve this problem at the moment. The situation pictured above still exists.

[MartyG-RealSense]

I conducted further research into face reconstruction with the D415. I see that you were also attempting to improve facial reconstructions in 2019.

https://github.com/IntelRealSense/librealsense/issues/4629

https://support.intelrealsense.com/hc/en-us/community/posts/360033162714-Can-3D-reconstruction-be-smoother-

The discussion in the link below that involves face reconstruction with a D415 may be helpful if you have not seen it already.

https://github.com/IntelRealSense/librealsense/issues/4422

[nevynwong]

I have encountered the problem here before and solved it before, but now I am not getting the same problem solved using these methods.

[MartyG-RealSense]

What was the method that you used before that no longer works for you, please?

[nevynwong]

depth unit to 100, the effect of a-factor to 0.18, disparity Shift is 40. I also adjusted the spatial filter and the temporal filter.

[MartyG-RealSense]

I performed further testing with the D415. I found that the setting that made the most difference to a close-up depth image of the face was increasing the Disparity Shift to '100' instead of 40. This will exclude the background but may improve the facial detail.

[nevynwong]

It doesn't matter whether the background is excluded or not, what I want is a good facial imaging result, but it seems that I can't solve this problem with the face no matter how I try.

[MartyG-RealSense]

@nevynwong I researched your problem again from the beginning. If you have not tried it already, it may be worth exporting a ply file from the RealSense SDK, importing it into the free open-source MeshLab 3D modelling software and applying simplification controls to the point cloud before converting it to a solid mesh.

This process is described in the guide linked to below for converting a RealSense ply into a solid model, under the heading Simplification / Sampling.

https://www.andreasjakl.com/capturing-3d-point-cloud-intel-realsense-converting-mesh-meshlab/

[nevynwong]

Hello, I am using this tool is the same result, I am now recalibrating the camera and test again. This effect should not have anything to do with the high precision or whatever inside the preset. Does the light in the environment have an effect on the results? I used to use LED lighting, now just in indoor lighting.

[MartyG-RealSense]

Hi @nevynwong If fluorescent lights such as ceiling strip lights are used in the indoor location where the camera is then this has the potential to introduce noise into the image due to the lights flickering at a rate that is difficult to see with the human eye. Using an FPS speed close to the operating frequency of the lights may help to reduce the impact. For some lights this may be 30 FPS and for others it may be 60 FPS.

Some models of fluorescent light that are found in regions such as Europe may use a power line frequency of 50 hz though, making it more difficult to match the camera FPS to.

[nevynwong]

Is this also related to the lighting? Even if I set the frame rate to 30, it still appears.

[MartyG-RealSense]

I think this is just a consequence of the 400 Series stereo depth hardware compared to the SR300's structured light technology. There is not anything else that I can suggest other than what has already been covered during this case, as my research has indicated that these results for solid facial scans seem typical for the 400 Series.

[nevynwong]

I've tried my best to avoid this problem, but I'm still running into it.So the problem is unavoidable, right?

[MartyG-RealSense]

I agree that you have done the best that you can. I do not know of anything else that can be done to enhance this result unfortunately.

[MartyG-RealSense]

Hi @nevynwong Do you wish to continue with this case please? Thanks!