search

oppo-us-research / SpacetimeGaussians

[CVPR 2024] Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis
https://oppo-us-research.github.io/SpacetimeGaussians-website/
Other
616 stars 45 forks source link

Query regarding dataset without ground truth poses #7

Closed jayaramreddy10 closed 10 months ago

jayaramreddy10 commented 10 months ago

Hi, a quick question regarding the limitations

In preprocessing step i.e pre_n3d.py script, it expects ground truth poses (poses_bounds.npy) in convertdynerftocolmapdb function to generate folders .... for Neural3DVideo dataset. Does the method handle datasets which don't have any ground truth extrinsics and intrinsics , basically just the videos from multiple viewpoints?

If it can handle those datasets, could you please suggest the changes required in the script?

Thanks

lizhan17 commented 10 months ago

Hi,you can use software like colmap to estimate the poses of videos if you don't have ground truth poses. (We recommend that you use Colmap conventions for poses.) We recommend you use the technicolor dataset loader for real world data, as images are uncentered. Our codebase mainly work with Colmap's sparse model output. You may create a new dataset loader from the technicolor dataset loader. the image file location and name in the data loader will be modified.

jayaramreddy10 commented 10 months ago

Ok, to be sure, you suggest to run colmap sparse model on images from all cameras at every timestep/offset?

lizhan17 commented 10 months ago

Ok, to be sure, you suggest to run colmap sparse model on images from all cameras at every timestep/offset?

I suggest that you can get poses from a single time stamp then use that estimated poses to get points for the rest time stamp. In such way, they can share the same world mapping at different time.

jayaramreddy10 commented 10 months ago

Ok, to be sure, you suggest to run colmap sparse model on images from all cameras at every timestep/offset?

I suggest that you can get poses from a single time stamp then use that estimated poses to get points for the rest time stamp. In such way, they can share the same world mapping at different time.

Agree but could you please elaborate this (use that estimated poses to get points for the rest time stamp) and how do we go about that?

So, lets say, I have generated, how do we use that to generate .... ?

lizhan17 commented 10 months ago

treat colmap_0's extrinsics and intrinsics as ground truth extrinsics and intrinsics. then use that ground truth extrinsics and intrinsics for the rest times.