How to obtain good depth maps or confidence maps while streaming?

[sandeepnmenon]

Thank you for the amazing package, I have been able to use my Iphone to stream rgb and depth thanks to this to my ROS system. I have a few questions on the Record3DStream api..

How to get the confidence map while streaming? I see that when I export the r3d format, I get the confidence map. The point cloud I construct from the exported data files is vastly better than the one from the data in the stream. I am suspecting this because I am not using confidence maps in the stream. How to get that? Or is there any other thing to be considered while streaming.

Is there any other way that I can improve the quality of the pointclouds from the streaming data.

[marek-simonik]

Firstly, thank you for using Record3D. Unfortunately, there is no way to obtain confidence maps while streaming. The depth maps you get via USB streaming and the ones stored in r3d files should be identical, so there should be no difference in point cloud quality (when constructing point clouds from depth maps alone). But I guess you meant that the point clouds you construct from offline data are better because you filter points with low confidence score, right?

(P.S. I did not reply to your comment https://github.com/marek-simonik/record3d/issues/39#issuecomment-1477315794 as it seems you asked the same questions in this issue.)

[sandeepnmenon]

Thank you @marek-simonik for your reply. Yes you are right, I am using confidence values on the depth. Even when I am exporting the dataset as ply, the individual point clouds are better than the streamed ones. Could you suggest any post processing that I can do on my streamed depth to get better point clouds. Also, how do you calculate the confidence values, I would like to see if its possible to do that while streaming and possibly add as a feature.

[marek-simonik]

The confidence values are provided by Apple's API (AFAIK, Apple does not describe how exactly are the values computed), not by me. I apologize, I am not sure which specific noise-filtering algorithms to recommend.

[sandeepnmenon]

Thank you @marek-simonik Closing the issue