search

spla-tam / SplaTAM

SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (CVPR 2024)
https://spla-tam.github.io/
BSD 3-Clause "New" or "Revised" License
1.58k stars 174 forks source link

Query regarding Keyframe Selection #62

Closed Buffyqsf closed 9 months ago

Buffyqsf commented 10 months ago

I found a little bug of your keyframe selecting strategy, as shown below. You sorted first, but then permuted before selecting k frames. It will make the sorting useless? I found some keyframes not reasonable in experiments also. image

Buffyqsf commented 9 months ago

I also found that after "solving this problem" as below, it seems that the scene is troubled by overfitting. Past frames tend to perform bad in the final eval. image

Nik-V9 commented 9 months ago

Hi, Thanks for the deep dive into the code!

The functionality of the code is correct. The sort operation is meaningless (it can be safely removed to make the code more optimized). My intention is to select k random keyframes with any points from the current frame projecting onto them.

The online overfitting behavior that you are seeing when selecting the top k keyframes is expected and is what we have seen. Generally, this happens because you choose frames that look very similar to the current frame (i.e., minimal baseline between images). However, for Gaussian Splatting, it is beneficial to select frames that look at similar things from distinct perspectives, i.e., a relatively large baseline or opposing viewpoints with overlap in scene content (this tends to happen when you start to have loop closures, as seen in Replica Room 0).