search

nerfstudio-project / nerfstudio

A collaboration friendly studio for NeRFs
https://docs.nerf.studio
Apache License 2.0
9.28k stars 1.25k forks source link

Postshot has better quality splats - due to suboptimal colmap usage/algorithm/both? #3421

Open tangfucius opened 1 week ago

tangfucius commented 1 week ago

TL;DR I compared the splats of a room generated by Splatfacto and Postshot with the same inputs, and consistently found Postshot to be crisper.

Do we already know if some components (ns-process-data or Splatfacto algorithm) are suboptimal, particularly regarding how ns-process-data uses colmap outputs?

Set up Inputs (for the same room):

Both Nerfstudio and Postshot were run on Windows.

Nerfstudio I used ns-process-data video/images to generate inputs for training.

  1. iPhone video: 364 images were extracted for the video, while transforms.json contains 342 frames.
  2. android: transforms.json only contains 25 frames (out of 56), even if I used --matching-method exhaustive.

Postshot I tried these 2 settings:

  1. Import the raw video/images directly, and let Postshot register the camera poses (Use Best Images for image selection)
    • For iPhone, it tracked 219 images from the video
    • For android, it tracked 52/56 images
  2. Use the cameras.bin + images.bin + points3D.bin + images folder generated from ns-process-data above
    • For iPhone, it managed to track 362/364 images
    • For android, it managed to track 47/56 images

Default settings were used for training (Splat MCMC). Subjectively, I found the splat quality of these 2 settings to be pretty similar.

Results Postshot tend to have crisper results for all cases. All renders are 1920 x 1080.

iPhone Nerfstudio iphone_caps_nerfstudio Postshot (direct import) iphone_raw0000 Postshot (colmap) iphone_colmap0000

android Nerfstudio redmi_colmap_nerfstudio Postshot (direct import) redmi_raw0000 Postshot (colmap) redmi_colmap0000

Questions As a supporter of open source, I'd love to see Nerfstudio match the quality of Postshot. Asking these questions to better understand where I can possibly contribute:

  1. Postshot matches more images using the same colmap outputs (the *.bin files) as Nerfstudio (transforms.json). Why is that the case?
  2. Would hcol perform better than colmap for ns-process-data? Tried using it but haven't managed to get the pycolmap dependency to install on Windows so far.
  3. Is the Splat MCMC algorithm used in Postshot an improved version of the original GS, or are they comparable? Would like to understand whether the quality difference mainly comes from SfM or the splatting algorithm - seems to be the former based on what I have seen so far.
jb-ye commented 1 week ago

Your iphone results doesn't look right to me. If you can share cameras.bin + images.bin + points3D.bin + images folder, I might be able to debug a bit.

tangfucius commented 1 week ago

Thanks for your response! I uploaded them here.

Is there anything else you'd suggest me to look at on my side? Bottom line, given the same inputs, do we expect the quality of Splatfacto to be similar to other implementations?

ichsan2895 commented 2 days ago

Hello @tangfucius , it is not fair comparison since splatfacto uses some images for eval dataset and not all images trained.

You can maximize the power of Splatfacto by using all images for training.

Try to add --eval-mode fraction --train-split-fraction 1 flags.

For example:

ns-train splatfacto-big nerfstudio-data --data path/to/dataset --eval-mode fraction --train-split-fraction 1