Is the Gaussian Splatting Nerfstudio export README accurate ?

orgicus commented 9 months ago

I've just tried to follow the mapping README and most of it worked!

I'm stuck at the last step:

Train as ns-train gaussian-splatting --data /example/output/path/my-nerf

I'm getting an error here:

ns-train gaussian-splatting --data sdk-examples/mapping/spectacular-ai-nerfstudio --vis viewer
usage: ns-train [-h]
                {depth-nerfacto,dnerf,generfacto,gs,instant-ngp,instant-ngp-bo
unded,mipnerf,nerfacto,nerfacto-big,nerfacto-huge,neus,neus-facto,phototourism
,semantic-nerfw,tensorf,vanilla-nerf,volinga,in2n,in2n-small,in2n-tiny,kplanes
,kplanes-dynamic,lerf,lerf-big,lerf-lite,nerfplayer-nerfacto,nerfplayer-ngp,te
tra-nerf,tetra-nerf-original}

+- Parsing error -------------------------------------------------------------+
| Argument                                                                    |
| {depth-nerfacto,dnerf,generfacto,gs,instant-ngp,instant-ngp-bounded,mipnerf |
| ,nerfacto,nerfacto-big,nerfacto-huge,neus,neus-facto,phototourism,semantic- |
| nerfw,tensorf,vanilla-nerf,volinga,in2n,in2n-small,in2n-tiny,kplanes,kplane |
| s-dynamic,lerf,lerf-big,lerf-lite,nerfplayer-nerfacto,nerfplayer-ngp,tetra- |
| nerf,tetra-nerf-original}: invalid choice: 'gaussian-splatting' (choose     |
| from 'depth-nerfacto', 'dnerf', 'generfacto', 'gs', 'instant-ngp',          |
| 'instant-ngp-bounded', 'mipnerf', 'nerfacto', 'nerfacto-big',               |
| 'nerfacto-huge', 'neus', 'neus-facto', 'phototourism', 'semantic-nerfw',    |
| 'tensorf', 'vanilla-nerf', 'volinga', 'in2n', 'in2n-small', 'in2n-tiny',    |
| 'kplanes', 'kplanes-dynamic', 'lerf', 'lerf-big', 'lerf-lite',              |
| 'nerfplayer-nerfacto', 'nerfplayer-ngp', 'tetra-nerf',                      |
| 'tetra-nerf-original')                                                      |
+-----------------------------------------------------------------------------+
C:\blip\nerfstudio\nerfstudio\utils\misc.py:184: RuntimeWarning: Windows does not yet support torch.compile and the performance will be affected.
  warnings.warn(
(nerfstudio)

I thought perhaps it's a type and also tried:

ns-train gaussiansplatting --data sdk-examples/mapping/spectacular-ai-nerfstudio --vis viewer
ns-train gaussian_splatting --data sdk-examples/mapping/spectacular-ai-nerfstudio --vis viewer but I get the same error.

I have installed gsplat in the same nerfstudio conda environment. I did notice gs and tried that too, but I get different errors:

ns-train gs --data sdk-examples/mapping/spectacular-ai-nerfstudio --vis viewer[15:31:31] Using --data alias for --data.pipeline.datamanager.data                                          train.py:231
Config
TrainerConfig(
    _target=<class 'nerfstudio.engine.trainer.Trainer'>,
    output_dir=WindowsPath('outputs'),
    method_name='gaussian_splatting',
    experiment_name=None,
    project_name='nerfstudio-project',
    timestamp='2023-11-21_153131',
    machine=MachineConfig(seed=42, num_devices=1, num_machines=1, machine_rank=0, dist_url='auto', device_type='cuda'),
    logging=LoggingConfig(
        relative_log_dir=WindowsPath('.'),
        steps_per_log=10,
        max_buffer_size=20,
        local_writer=LocalWriterConfig(
            _target=<class 'nerfstudio.utils.writer.LocalWriter'>,
            enable=True,
            stats_to_track=(
                <EventName.ITER_TRAIN_TIME: 'Train Iter (time)'>,
                <EventName.TRAIN_RAYS_PER_SEC: 'Train Rays / Sec'>,
                <EventName.CURR_TEST_PSNR: 'Test PSNR'>,
                <EventName.VIS_RAYS_PER_SEC: 'Vis Rays / Sec'>,
                <EventName.TEST_RAYS_PER_SEC: 'Test Rays / Sec'>,
                <EventName.ETA: 'ETA (time)'>
            ),
            max_log_size=10
        ),
        profiler='basic'
    ),
    viewer=ViewerConfig(
        relative_log_filename='viewer_log_filename.txt',
        websocket_port=None,
        websocket_port_default=7007,
        websocket_host='0.0.0.0',
        num_rays_per_chunk=32768,
        max_num_display_images=512,
        quit_on_train_completion=False,
        image_format='jpeg',
        jpeg_quality=90
    ),
    pipeline=GaussianSplattingPipelineConfig(
        _target=<class 'nerfstudio.pipelines.gaussian_splatting_pipeline.GaussianSplattingPipeline'>,
        datamanager=GaussianSplattingDatamanagerConfig(
            _target=<class 'nerfstudio.data.datamanagers.gaussian_splatting_datamanager.GaussianSplattingDatamanager'>,
            data=WindowsPath('sdk-examples/mapping/spectacular-ai-nerfstudio'),
            camera_optimizer=None,
            masks_on_gpu=False,
            images_on_gpu=False
        ),
        model=GaussianSplattingModelConfig(
            _target=<class 'nerfstudio.models.gaussian_splatting.GaussianSplatting'>,
            enable_collider=True,
            collider_params={'near_plane': 2.0, 'far_plane': 6.0},
            loss_coefficients={'rgb_loss_coarse': 1.0, 'rgb_loss_fine': 1.0},
            eval_num_rays_per_chunk=4096,
            prompt=None,
            background_color='black',
            sh_degree=3
        )
    ),
    optimizers={},
    vis='viewer',
    data=WindowsPath('sdk-examples/mapping/spectacular-ai-nerfstudio'),
    prompt=None,
    relative_model_dir=WindowsPath('nerfstudio_models'),
    load_scheduler=True,
    steps_per_save=2000,
    steps_per_eval_batch=500,
    steps_per_eval_image=500,
    steps_per_eval_all_images=25000,
    max_num_iterations=30000,
    mixed_precision=True,
    use_grad_scaler=False,
    save_only_latest_checkpoint=True,
    load_dir=None,
    load_step=None,
    load_config=None,
    load_checkpoint=None,
    log_gradients=False,
    gradient_accumulation_steps=1
)

           Saving config to:                                                                    experiment_config.py:136
           outputs\spectacular-ai-nerfstudio\gaussian_splatting\2023-11-21_153131\config.yml
           Saving checkpoints to:                                                                         trainer.py:135
           outputs\spectacular-ai-nerfstudio\gaussian_splatting\2023-11-21_153131\nerfstudio_models
C:\blip\nerfstudio\nerfstudio\utils\misc.py:184: RuntimeWarning: Windows does not yet support torch.compile and the performance will be affected.
  warnings.warn(
Traceback (most recent call last):
  File "C:\Users\george.profenza\.pyenv\pyenv-win\versions\miniconda3\envs\nerfstudio\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\george.profenza\.pyenv\pyenv-win\versions\miniconda3\envs\nerfstudio\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\george.profenza\.pyenv\pyenv-win\versions\miniconda3\envs\nerfstudio\Scripts\ns-train.exe\__main__.py", line 7, in <module>
  File "C:\blip\nerfstudio\nerfstudio\scripts\train.py", line 261, in entrypoint
    main(
  File "C:\blip\nerfstudio\nerfstudio\scripts\train.py", line 246, in main
    launch(
  File "C:\blip\nerfstudio\nerfstudio\scripts\train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "C:\blip\nerfstudio\nerfstudio\scripts\train.py", line 99, in train_loop
    trainer.setup()
  File "C:\blip\nerfstudio\nerfstudio\engine\trainer.py", line 148, in setup
    self.pipeline = self.config.pipeline.setup(
  File "C:\blip\nerfstudio\nerfstudio\configs\base_config.py", line 54, in setup
    return self._target(self, **kwargs)
TypeError: __init__() missing 1 required positional argument: 'ref_orientation'
(nerfstudio)

As per args parsing error, I would expect gaussian-splatting to be an option, but it's not ? (nerfstudio does see other packages like volinga and that works just fine).

I appreciate this may be more of a nerfstudio/gsplat question than a SpectacularAI one: any hints/tips are more than welcome and I'd be more than happy to submit a PR to update documentation if that helps.

Thank you so much, George

oseiskar commented 9 months ago

Thank you for reporting! The instructions indeed had a critical flaw: they failed to specify that the gaussian-splatting branch of Nerfstudio is currently needed. Now updated. Should be installed like this:

git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
git checkout gaussian-splatting  # <--- Different from standard instructions
pip install --upgrade pip setuptools
pip install -e .

orgicus commented 9 months ago

That was it! Thank you so much for the notes and updating the guide! Awesome stuff!

SpectacularAI / sdk-examples

Is the Gaussian Splatting Nerfstudio export README accurate ? #99