Run only CamP and extract optimized camera poses

[kotetsu-n]

Hi, thank you for making such a great achievement public! This is not an issue, but a question. Can I run only CamP and obtain optimized camera poses? I quickly checked the code, but it seems deeply integrated ZipNeRF and CamP. If there are any options which allow me to do it, it would be great if you could share it. Thank you for your help in advance.

[ichsan2895]

Agree. It would be good if we can run Camp only. We interested for using it with many nerf method and gaussian splatting

[lanxinger]

+1 on this. I would be very curious to also see it on the ARKit scenes captured with the open source nerf app as was described in the CamP paper as it showed a big improvement over colmap.

Is there any plans or timeline to release those datasets?

[jonbarron]

CamP jointly solves for camera poses and radiance fields, so there's no way to use it to just run CamP without doing everything. You can save out the CamP camera poses after solving if you want. But if the goal here is to just get better camera poses as quickly as possible from some black box, I would just use COLMAP. We don't have any strong evidence that CamP outperforms slow offline SFM packages like COLMAP and the many other papers and codebases in this space, and my instinct is that the SOTA in that literature is better than CamP. The upside of CamP is primarily speed, not quality.

[pwais]

@jonbarron In the future might your team please consider evaluating on datasets that have ground truth poses / geometry? E.g. DTU or ETH3D would be easy here. Google clearly has the money and time (e.g. the projects leading up to Google Scanned Objects include DTU-like camera pose measurement, or even more accurate, though not necessarily released publicly). It's critically important as research on novel view synthesis and separately reconstruction (e.g. SDFs) continues to converge. You're not the only guilty party, e.g. Efros-branded "COLMAP-Free 3d Gaussian Splatting" ( https://arxiv.org/pdf/2312.07504.pdf ) also evaluates on Tanks & Temples, which is COLMAP and not actual ground truth poses. As artists and practitioners become more active engaging with this research, it is critical to communicate metrics besides MSE when making claims about geometry.

[Brunwo]

I've started running train configs configs/zipnerf/360.gin & configs/camp/camera_optim.gin on google colab free gpu tier A single scene with the default 200k iterations will take about 3 days, is that what I should be expecting ?

[jonbarron]

I've started running train configs configs/zipnerf/360.gin & configs/camp/camera_optim.gin on google colab free gpu tier A single scene with the default 200k iterations will take about 3 days, is that what I should be expecting ?

yeah that sounds plausible to me. I wouldn't expect the colab free GPUs to be particularly fast, and we train for a super long time by default.

[hadillivvy]

CamP jointly solves for camera poses and radiance fields, so there's no way to use it to just run CamP without doing everything. You can save out the CamP camera poses after solving if you want. But if the goal here is to just get better camera poses as quickly as possible from some black box, I would just use COLMAP. We don't have any strong evidence that CamP outperforms slow offline SFM packages like COLMAP and the many other papers and codebases in this space, and my instinct is that the SOTA in that literature is better than CamP. The upside of CamP is primarily speed, not quality.

Where can we get CamP camera poses in the train.py code?

[RR-28023]

We don't have any strong evidence that CamP outperforms slow offline SFM packages like COLMAP and the many other papers and codebases in this space, and my instinct is that the SOTA in that literature is better than CamP. The upside of CamP is primarily speed, not quality.

@jonbarron forgive my ignorance but would you mind clarifying how does your statement above reconcile with the results in the paper (e.g Table 1), where you outperform the results obtained with COLMAP? Are you suggesting that CamP may find camera parameters that better optimize the scene reconstruction (when combined with Zip-NeRF's MLP parameters), despite those not being necessarily closer to the ground truth parameters than those obtained with COLMAP?

[jonbarron]

Sorry, when I said that "CamP doesn't outperform COLMAP", I meant that CamP initialized to ARKit poses doesn't necessarily outperform COLMAP. CamP initialize to COLMAP does consistently outperform COLMAP, which is the baseline in Table 1.