gradeeterna
commented
1 month ago This would be great! I would like to use nerfacto with the camera optimization on, and then train splatfacto on the original dataset with the adjusted poses. To do this currently, you would need to manually convert the output from "ns-export cameras" to align with the original input data.
Is your feature request related to a problem? Please describe. Currently, it's possible to export (refined) camera poses from a trained model in json form. Unfortunately, the exported json is not directly compatible with the transforms.json input (image paths are changed to absolute, image order is changed, train and eval datasets are split and parameters required for transforms.json are omitted), so a lot of manual conversion needs to be done. Additionally, splatfacto relies on the colmap .bin files and pointcloud as well. If you only export the refined poses, modify them into a valid transforms.json, then try to train a splatfacto with the resulting poses, the initial pointcloud is misaligned, only random init is usable.
Describe the solution you'd like Ideally I would like a way to export a complete ns-data/colmap dataset, optionally including exporting an aligned pointcloud generated from our nerf/splat result (with configurable sparsity). Additionally, being able to simply export transforms.json in a compatible format directly would also be nice to have, but limited due to above outlined alignment issues.
Describe alternatives you've considered An external script could do it, of course, but it feels like it would be more useful to have it in Nerfstudio directly.
Additional context Use case: If you have a dataset with poor poses, you can optimize them by running a training run with pose optimization enabled (i.e. running a nerfacto-huge), but the output quality suffers during that run. Also, not all models might be equally good at pose optimization (i.e. nerfacto-huge might be better at it than splatfacto-big). "Pre-running" a training and exporting the poses/dataset can result in a significantly better dataset, and thus a significantly better result using other methods. Plus having the "fixed" dataset means it can be used in other projects as-is too.
I've tried to look into implementing it myself, but I don't know the Nerfstudio codebase well enough to make it happen (yet) with the limited time I have.