The accuracy of the depth camera

[Bornblack]

Hi! As for my project, I want to acquiring point cloud from depth camera for object 6DOF estimation. I find there is not much docs about the accuracy of depth camera. I use the code from https://github.com/petergu684/HoloLens2-ResearchMode-Unity then, visualize the point cloud in HoloLens. As for plane like the desk, the point cloud could align the plane well. when it comes to some stuff with complex surface, the misalignment is about 2cm under the surface. The paper below indicates the misalignment could be related to the Multi Path Effect of ToF sensor. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7070293/ Is there any way could solve this？

Any reply will be appreciated.

[kysucix]

hi, the multi path effect usually happens in specif part of the scenes such as the corner of a room. Could you provide an example of the misalignment you are experiencing. I'm not sure I've experienced such a systematic error.

[Bornblack]

Thanks for your reply! After some tests, I find the misalignment mainly happens on a 3D-Printing model. The printing material is photosensitive resin, and I guess the special material should be blame for the misalignment causing kind of complex physical phenomenon involving optics. I have also done some funny tests like reconstructing the point cloud of the face of my colleagues(holding still) and visualizing it in real time, there was also a slight misalignment .:joy: But I thinks the result is reasonable as the capability of the depth sensor.

Recently I encounter another problem . I normalize the raw AB frame data (16 bit-->8bits, 0~255) and visualize it. There are some strange values in the image as shown below. The strange stuff on the monitor is probably caused by the light source of depth sensor. But why dose some parts of my hand near the depth sensor look darker?

Any reply will be appreciated.

[agroenenberg]

I have the same problem. The pointcloud is projected about 2 cm under the real surface. XY are projected perfectly but the real depth seems to have some kind of systematic error. It's always in the Z/ depth direction as seen from the HoloLens 2. I used mattifying spray on my 3D print but on the 3D print as well as a human face there seems to be an offset

[chiarapalu]

Same here, however an optical tracking marker is almost perfectly detected in depth. Any suggestion on how to improve this aspect?Thanks!

[agroenenberg]

Any update on this @kysucix ? It would be nice to get this improved, but I feel like this git has gotten some radiosilence... :( The problem should be somewhere in the depth sensor or in the RMApi functions provided to extract the depth map, since the full scene meshes also have this offset on the z-axis while fully rendered as a mesh on the HoloLens 2. If I overlay my hologram onto the readily recorded pointcloud, it is exactly on the right location. Only when rendering via the HoloLens 2, the offset @Bornblack describes is visible.

[petergu684]

So, I know it has been a while. Sorry that I do not take too much attention to issues here. I found this issue on HoloLens 1 and when HoloLens 2 was just released, I wrote this project and test again, only to find the issue still exists on this device. We even published a workshop paper on this issue before. It should be a general problem for ToF camera because of how it works. The accuracy of depth is actually great on some material that are good at reflecting infrared light such as glass bead, which is used for surgical retro-reflective marker. For this reason, we have another project to track retro-reflective sphere using the IR camera in short-throw mode here, and it works reasonably well (to me) with the addition of Kalman filter. 3D printed object in general does not have good accuracy. Since the error is systematic, an easy fix is that if accuracy is not of utmost importance in your application (say <1cm accuracy is good enough), and you know what material you are looking at, you can estimate the offset and apply offset manually. There is another paper recently that tried to solve this problem on some specific application using more advanced/customized calibration method and TSDF. That paper also covered and analyzed this depth sensor error if you are interested. If anyone has better idea how to solve it more elegantly, feel free to leave comment here or publish a paper :)