search

spla-tam / SplaTAM

SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (CVPR 2024)
https://spla-tam.github.io/
BSD 3-Clause "New" or "Revised" License
1.58k stars 174 forks source link

fix saving checkpoint at every iteration #37

Closed spirobel closed 10 months ago

spirobel commented 11 months ago

saving a checkpoint for every iteration only worked for me after removing time_idx % from this line

I used this in conjunction with: https://github.com/spla-tam/SplaTAM/pull/29 to make ply files that I was able to view with https://playcanvas.com/super-splat

Thanks for the great work!

Nik-V9 commented 11 months ago

Hi, Thanks for trying out our code! This is not a bug, and checkpointing works correctly on our end. For checkpointing, you need to update the save_checkpoints and checkpoint_interval parameters in the following example config file: https://github.com/spla-tam/SplaTAM/blob/bbaf5cc5754bf1034b33902007872c694e412a31/configs/iphone/splatam.py#L46

spirobel commented 11 months ago

I used the splatam.py in the replica config dir (I don't have an iphone) To get a checkpoint on every iteration I set 

    save_checkpoints=True, # Save Checkpoints
    checkpoint_interval=0, # Checkpoint Interval

I used this as a starting point because I want to feed stable diffusion images (with their respective depth images) into splatam and just wanted to check how splatam reacts to a single image and if the splats are good.

Do you have any thoughts on making this work with marigold depth images?

The iphone produces absolute depth information while marigold is relative, right? ... not sure yet if I should start hacking based on this code base or if it is better to start from scratch ...

The end goal is to pick camera positions and let stable diffusion inpaint something (then let splatam embed it into the splat) then pick the next position and therefore iteratively make a 3d scene with stable diffusion and gaussian splats.

rosejn commented 10 months ago

@spirobel it's using modulo arithmetic so if you want a checkpoint every iteration you need to set checkpoint_interval to 1, not zero.

rosejn commented 10 months ago

oh yeah, if you have relative depth rather than absolute you might want to investigate using the pearson correlation coefficient computed over the depth prediction from the model and your depth map. I've seen some work that uses PCC over the full depth and others that do it patchwise.

Nik-V9 commented 10 months ago

Hi, as @rosejn mentioned, the checkpoint_interval needs to be set to 1 (it can't be 0) (this is not specific to the iPhone config file; it works with any other config file).

The experiment sounds interesting! We have been looking at PatchFusion monocular depth for Outdoor SplaTAMs (video shared below).

Please open a new issue to discuss using monocular relative depth images for SplaTAM.

https://github.com/spla-tam/SplaTAM/assets/56788999/1e68dab0-12c9-44a3-b027-ff4f1f6a2601