Training with a live streaming

spla-tam / SplaTAM

SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (CVPR 2024)

https://spla-tam.github.io/

BSD 3-Clause "New" or "Revised" License

1.35k stars 150 forks source link

Training with a live streaming #105

Open jinn9403 opened 2 months ago

jinn9403 commented 2 months ago

Really good research and thanks for sharing the source. Especially the reconstruction using Gaussian splatting in fast time seems to be a step closer to real-time reconstruction.

When doing online training, I noticed that instantNGP has a function to show the training process while receiving streaming via nerfcapture app. You used the dataset_capture_loop function to stream the dataset to a folder and train based on it. Is it possible to apply the def live_streaming_loop function to train in real time?

And the output is supposed to be viewable in open3d. If I want to view it in nerfstudio viewer or any other viewer, please let me know where to modify it.

Nik-V9 commented 3 weeks ago

Thanks!

Yes, updating the code to show live training should be possible. However, we currently don't have this implemented. Furthermore, SplaTAM needs further optimization to enable faster reconstruction.

Regarding other viewers for Gaussian Splats: We support exporting our output to the standard .ply format used by various viewers like SuperSplat & PolyCam (not sure what NeRFStudio uses):

python scripts/export_ply.py configs/...