search

princeton-vl / RAFT-Stereo

MIT License
721 stars 139 forks source link

confidence map #59

Closed gpuartifact closed 2 years ago

gpuartifact commented 2 years ago

Hello, Is it possible to extract a confidence map for all disparity values? During a 3D stiching and SLAM such confidence map would greatly improve the over all result. Do you have any suggestions on how to extract such confidence map from the RAFT pipeline itself or would you recommend a different approach? Thank you for making your work available to all!

lahavlipson commented 2 years ago

Our method does not explicitly predict a confidence map.

However, in order to produce the reconstructions from the README, we remove the "flying points" by only including pixels with a small disparity difference (<1) from their neighbor. This seems to work pretty well.

Another option is to use forward backward consistency. The correlation magnitudes may also show the confidence of each prediction, but I haven't tried this.

gpuartifact commented 2 years ago

Correltation magnitudes is a good hint, thanks! I am currently struggling with specular and specle like reflections which result in flatter depth maps due to poor correlation between the stereo images in higher spacial frequencies. Interestingly, training with simulation data that also contains specular reflections improves the results regarding outliers but of course still results in overall flatter depth maps. forward backward consitency is also helpfull, especially for occlusion. Thanks