search

nerfstudio-project / nerfstudio

A collaboration friendly studio for NeRFs
https://docs.nerf.studio
Apache License 2.0
8.87k stars 1.18k forks source link

Splatfacto in the wild #3201

Closed KevinXu02 closed 4 days ago

KevinXu02 commented 3 weeks ago

Implimentation of splatfacto in the wild and 2d background model for splatfacto. https://kevinxu02.github.io/gsw.github.io/

ichsan2895 commented 3 weeks ago

This PR needs gsplat==1.0.0 or gsplat==0.1.12?

KevinXu02 commented 3 weeks ago

This PR needs gsplat==1.0.0 or gsplat==0.1.12?

Currently it uses gsplat==0.1.12. I'll update it to gsplat==1.0.0 when I got time.

ichsan2895 commented 3 weeks ago

I think it is better If your PR pin specific gsplat and nerfstudio since the latest commit nerfstudio, it already pin gsplat==1.0.0.

When it merges to main nerfstudio repo, so it will use gsplat==1.0.0.

My idea, pinned your repo with nerfstudio==1.1.0 and gsplat==0.1.12

KevinXu02 commented 3 weeks ago

I think it is better If your PR pin specific gsplat and nerfstudio since the latest commit nerfstudio, it already pin gsplat==1.0.0.

When it merges to main nerfstudio repo, so it will use gsplat==1.0.0.

My idea, pinned your repo with nerfstudio==1.1.0 and gsplat==0.1.12

Thank you for your advice! I guess update this to gsplat==1.0.0 won't be super hard. 1.0 version of gsplat reduced the gram usage a lot thus making this new method can be ran on more devices. And the bg model could also be useful for newer version of nerfstudio.

ichsan2895 commented 3 weeks ago

Good luck 👍

BTW, why the default --steps-per-eval-all-images and --steps-per-eval-image are set to 1000000 ?

Is there any problem if I lower it?

KevinXu02 commented 3 weeks ago

Good luck 👍

BTW, why the default --steps-per-eval-all-images and --steps-per-eval-image are set to 1000000 ?

Is there any problem if I lower it?

Because the eval for nerf in the wild uses left half eval images for training and another half for eval. I need to sort of hack the dataloader to make it work but it's ugly and not included in this pr. It won't be a problem if you lower it, but it will always uses 0th image's appearance embedding for output.

ichsan2895 commented 2 weeks ago

I got error when I exported the splat

>>> ns-export gaussian-splat --load-config outputs/unnamed/splatfacto-w/2024-06-10_092617/config.yml --output-dir renders/MY_DATASET_SPLATWILD/

WARNING: Using slower TCNN CutlassMLP instead of TCNN FullyFusedMLP
Use layer width of 16, 32, 64, or 128 to use the faster TCNN FullyFusedMLP.

Loading latest checkpoint from load_dir
✅ Done loading checkpoint from outputs/unnamed/splatfacto-w/2024-06-10_092617/nerfstudio_models/step-000064999.ckpt
Traceback (most recent call last):
  File "/usr/local/bin/ns-export", line 8, in <module>
    sys.exit(entrypoint())
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 638, in entrypoint
    tyro.cli(Commands).main()
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 550, in main
    assert isinstance(pipeline.model, SplatfactoModel)
AssertionError

additional information

RTX A6000 48 GB
Ubuntu 22.04.3 LTS
python 3.10.12
torch 2.0.1+cu118
https://github.com/KevinXu02/nerfstudio/tree/splatfactow@eb9ee7a0aa7ac31ad768b5a5ac1b7eeae4af10ef
gsplat==0.1.12
KevinXu02 commented 2 weeks ago

I got error when I exported the splat

>>> ns-export gaussian-splat --load-config outputs/unnamed/splatfacto-w/2024-06-10_092617/config.yml --output-dir renders/MY_DATASET_SPLATWILD/

WARNING: Using slower TCNN CutlassMLP instead of TCNN FullyFusedMLP
Use layer width of 16, 32, 64, or 128 to use the faster TCNN FullyFusedMLP.

Loading latest checkpoint from load_dir
✅ Done loading checkpoint from outputs/unnamed/splatfacto-w/2024-06-10_092617/nerfstudio_models/step-000064999.ckpt
Traceback (most recent call last):
  File "/usr/local/bin/ns-export", line 8, in <module>
    sys.exit(entrypoint())
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 638, in entrypoint
    tyro.cli(Commands).main()
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 550, in main
    assert isinstance(pipeline.model, SplatfactoModel)
AssertionError

additional information

RTX A6000 48 GB
Ubuntu 22.04.3 LTS
python 3.10.12
torch 2.0.1+cu118
https://github.com/KevinXu02/nerfstudio/tree/splatfactow@eb9ee7a0aa7ac31ad768b5a5ac1b7eeae4af10ef
gsplat==0.1.12

I got error when I exported the splat

>>> ns-export gaussian-splat --load-config outputs/unnamed/splatfacto-w/2024-06-10_092617/config.yml --output-dir renders/MY_DATASET_SPLATWILD/

WARNING: Using slower TCNN CutlassMLP instead of TCNN FullyFusedMLP
Use layer width of 16, 32, 64, or 128 to use the faster TCNN FullyFusedMLP.

Loading latest checkpoint from load_dir
✅ Done loading checkpoint from outputs/unnamed/splatfacto-w/2024-06-10_092617/nerfstudio_models/step-000064999.ckpt
Traceback (most recent call last):
  File "/usr/local/bin/ns-export", line 8, in <module>
    sys.exit(entrypoint())
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 638, in entrypoint
    tyro.cli(Commands).main()
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 550, in main
    assert isinstance(pipeline.model, SplatfactoModel)
AssertionError

additional information

RTX A6000 48 GB
Ubuntu 22.04.3 LTS
python 3.10.12
torch 2.0.1+cu118
https://github.com/KevinXu02/nerfstudio/tree/splatfactow@eb9ee7a0aa7ac31ad768b5a5ac1b7eeae4af10ef
gsplat==0.1.12

I got error when I exported the splat

>>> ns-export gaussian-splat --load-config outputs/unnamed/splatfacto-w/2024-06-10_092617/config.yml --output-dir renders/MY_DATASET_SPLATWILD/

WARNING: Using slower TCNN CutlassMLP instead of TCNN FullyFusedMLP
Use layer width of 16, 32, 64, or 128 to use the faster TCNN FullyFusedMLP.

Loading latest checkpoint from load_dir
✅ Done loading checkpoint from outputs/unnamed/splatfacto-w/2024-06-10_092617/nerfstudio_models/step-000064999.ckpt
Traceback (most recent call last):
  File "/usr/local/bin/ns-export", line 8, in <module>
    sys.exit(entrypoint())
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 638, in entrypoint
    tyro.cli(Commands).main()
  File "/workspace/NERFSTUDIO_SPLATWILD/nerfstudio/nerfstudio/scripts/exporter.py", line 550, in main
    assert isinstance(pipeline.model, SplatfactoModel)
AssertionError

additional information

RTX A6000 48 GB
Ubuntu 22.04.3 LTS
python 3.10.12
torch 2.0.1+cu118
https://github.com/KevinXu02/nerfstudio/tree/splatfactow@eb9ee7a0aa7ac31ad768b5a5ac1b7eeae4af10ef
gsplat==0.1.12

I haven't added support for the export yet bc both the gs colors and the background are implicit and need to be specially handled when exporting. But you should be able to view the result in ns-viewer.

And for splatfacto the gaussians should can be correctly exported but the background should be lost if you enables it. I'm planning to export it as a sky box but the current viewer doesn't support rendering sky boxes.

Thank you for telling me this issue!

ichsan2895 commented 2 weeks ago

I think you can create splat for background scene which is shaped like sphere. The color is the interpolation of boundary of current splat. This idea is inspired from Luma AI.

Here is the outside of the sphere image

Here is the inside of the sphere. I take this scene on (almost) the top of a mountain. image

KevinXu02 commented 2 weeks ago

It supports gsplat 1.0 now and I keep the cull to bigs as the result would looks better. This also includes a refined background logic.

KevinXu02 commented 2 weeks ago

Hi! This pr is ready for review @kerrj @brentyi

kerrj commented 4 days ago

Closing this since @KevinXu02 is planning on making a template repo!