search

HengyiWang / spann3r

3D Reconstruction with Spatial Memory
https://hengyiwang.github.io/projects/spanner
626 stars 28 forks source link

Freeze or Finetune dust3r #27

Closed Svithjod closed 1 hour ago

Svithjod commented 2 hours ago

Hi! Thanks for your excellent work!

I'm training the model on Co3d dataset, and I found you did not freeze dust3r model. My question is, is it important to finetune the dust3r model along with the memory network? What if I freeze the dust3r and train the memory network only?

HengyiWang commented 1 hour ago

Hi @Svithjod, I didn't try freezing dust3r. My intuition is that it will not work well because spann3r only reads from memory once to save computational costs. However, you may want to try freezing the ViT encoder to save the computational cost. I didn't freeze any parts mostly because at the beginning, I thought training on multiple frames could let the model learn a more robust feature compared to two frames only.

Svithjod commented 1 hour ago

OK I'll try it out, thank you!