InstructGS2GSPipeline Not Used During Training

[StephenYangjz]

Hi, it seems like with the most up to date igs2gs, gsplat, and nerfstudio repo, training a gs2gs isnt fully working on my end.

The command I use is ns-train igs2gs --data conditioning/gen_dataset --load-dir outputs/unnamed/splatfacto/2024-04-15_221736/nerfstudio_models --pipeline.model.prompt "turn the room into red color" --pipeline.guidance-scale 12.5 --pipeline.image-guidance-scale 1.5

It first doesnt recognize the guidance scale arguments (and they are not even in ns-train igs2gs --help)

If I remove the argument and train it when just running ns-train igs2gs --data conditioning/gen_dataset --load-dir outputs/unnamed/splatfacto/2024-04-15_221736/nerfstudio_models --pipeline.model.prompt "turn the room into red color", it would rather be not running the InstructGS2GSPipeline at all. After training the gs isnt changed, and I dong think the InstructGS2GSPipeline gets run after some playing around.

Maybe there's some hints printed out below

InstructGS2GSTrainerConfig(
    _target=<class 'igs2gs.igs2gs_trainer.InstructGS2GSTrainer'>,
    output_dir=PosixPath('outputs'),
    method_name='igs2gs',
    experiment_name=None,
    project_name='nerfstudio-project',
    timestamp='2024-04-16_000500',
    machine=MachineConfig(seed=42, num_devices=1, num_machines=1, machine_rank=0, dist_url='auto', device_type='cuda'),
    logging=LoggingConfig(
        relative_log_dir=PosixPath('.'),
        steps_per_log=10,
        max_buffer_size=20,
        local_writer=LocalWriterConfig(
            _target=<class 'nerfstudio.utils.writer.LocalWriter'>,
            enable=True,
            stats_to_track=(
                <EventName.ITER_TRAIN_TIME: 'Train Iter (time)'>,
                <EventName.TRAIN_RAYS_PER_SEC: 'Train Rays / Sec'>,
                <EventName.CURR_TEST_PSNR: 'Test PSNR'>,
                <EventName.VIS_RAYS_PER_SEC: 'Vis Rays / Sec'>,
                <EventName.TEST_RAYS_PER_SEC: 'Test Rays / Sec'>,
                <EventName.ETA: 'ETA (time)'>
            ),
            max_log_size=10
        ),
        profiler='basic'
    ),
    viewer=ViewerConfig(
        relative_log_filename='viewer_log_filename.txt',
        websocket_port=None,
        websocket_port_default=7007,
        websocket_host='0.0.0.0',
        num_rays_per_chunk=32768,
        max_num_display_images=512,
        quit_on_train_completion=False,
        image_format='jpeg',
        jpeg_quality=75,
        make_share_url=False,
        camera_frustum_scale=0.1,
        default_composite_depth=True
    ),
    pipeline=InstructGS2GSPipelineConfig(
        _target=<class 'nerfstudio.pipelines.base_pipeline.VanillaPipeline'>,
        datamanager=InstructGS2GSDataManagerConfig(
            _target=<class 'nerfstudio.data.datamanagers.full_images_datamanager.FullImageDatamanager'>,
            data=PosixPath('conditioning/gen_dataset'),
            masks_on_gpu=False,
            images_on_gpu=False,
            dataparser=BlenderDataParserConfig(
                _target=<class 'nerfstudio.data.dataparsers.blender_dataparser.Blender'>,
                data=PosixPath('data/blender/lego'),
                scale_factor=1.0,
                alpha_color='white',
                ply_path=None
            ),
            camera_res_scale_factor=1.0,
            eval_num_images_to_sample_from=-1,
            eval_num_times_to_repeat_images=-1,
            eval_image_indices=(0,),
            cache_images='cpu',
            cache_images_type='float32',
            max_thread_workers=None
        ),
        model=InstructGS2GSModelConfig(
            _target=<class 'igs2gs.igs2gs.InstructGS2GSModel'>,
            enable_collider=True,
            collider_params={'near_plane': 2.0, 'far_plane': 6.0},
            loss_coefficients={'rgb_loss_coarse': 1.0, 'rgb_loss_fine': 1.0},
            eval_num_rays_per_chunk=4096,
            prompt='turn the room into red color',
            warmup_length=500,
            refine_every=100,
            resolution_schedule=3000,
            background_color='random',
            num_downscales=2,
            cull_alpha_thresh=0.1,
            cull_scale_thresh=0.5,
            continue_cull_post_densification=True,
            reset_alpha_every=30,
            densify_grad_thresh=0.0002,
            densify_size_thresh=0.01,
            n_split_samples=2,
            sh_degree_interval=1000,
            cull_screen_size=0.15,
            split_screen_size=0.05,
            stop_screen_size_at=4000,
            random_init=False,
            num_random=50000,
            random_scale=10.0,
            ssim_lambda=0.2,
            stop_split_at=15000,
            sh_degree=3,
            use_scale_regularization=False,
            max_gauss_ratio=10.0,
            output_depth_during_training=False,
            rasterize_mode='classic',
            camera_optimizer=CameraOptimizerConfig(
                _target=<class 'nerfstudio.cameras.camera_optimizers.CameraOptimizer'>,
                mode='off',
                trans_l2_penalty=0.01,
                rot_l2_penalty=0.001,
                optimizer=None,
                scheduler=None
            ),
            use_lpips=True,
            use_l1=True,
            patch_size=32,
            lpips_loss_mult=1.0
        )
    ),
    optimizers={
        'means': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.00016,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': ExponentialDecaySchedulerConfig(
                _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                lr_pre_warmup=1e-08,
                lr_final=1.6e-06,
                warmup_steps=0,
                max_steps=30000,
                ramp='cosine'
            )
        },
        'features_dc': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.0025,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        },
        'features_rest': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.000125,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        },
        'opacities': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.05,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        },
        'scales': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.005,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        },
        'quats': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.001,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': None
        },
        'camera_opt': {
            'optimizer': AdamOptimizerConfig(
                _target=<class 'torch.optim.adam.Adam'>,
                lr=0.001,
                eps=1e-15,
                max_norm=None,
                weight_decay=0
            ),
            'scheduler': ExponentialDecaySchedulerConfig(
                _target=<class 'nerfstudio.engine.schedulers.ExponentialDecayScheduler'>,
                lr_pre_warmup=1e-08,
                lr_final=5e-05,
                warmup_steps=0,
                max_steps=30000,
                ramp='cosine'
            )
        }
    },
    vis='viewer',
    data=PosixPath('conditioning/gen_dataset'),
    prompt=None,
    relative_model_dir=PosixPath('nerfstudio_models'),
    load_scheduler=True,
    steps_per_save=500,
    steps_per_eval_batch=100,
    steps_per_eval_image=100,
    steps_per_eval_all_images=100000,
    max_num_iterations=7500,
    mixed_precision=False,
    use_grad_scaler=False,
    save_only_latest_checkpoint=True,
    load_dir=PosixPath('outputs/unnamed/splatfacto/2024-04-15_221736/nerfstudio_models'),
    load_step=None,
    load_config=None,
    load_checkpoint=None,
    log_gradients=False,
    gradient_accumulation_steps={'camera_opt': 100, 'color': 10, 'shs': 10}
)

Thank you and any pointers would be greatly appreciated @cvachha @ayaanzhaque !

[cvachha]

Hi, I'm unsure why it might not be working on your project. Regarding the image guidance arguments, I'm not getting the unrecognized options error and I'm seeing them in the pipeline section of ns-train igs2gs --help. Could you try updating the repo with git pull and run pip install -e . in the igs2gs folder? You might also need to update nerfstudio, but I'm not sure.

The pipeline should be getting called including the Instruct-Pix2Pix image edit. Could you try saving and checking the instruct-pix2pix edited image to see if it is running correctly (adding plt.imsave("edited_image_"+ str(idx)+".jpg", edited_image.squeeze().permute(1, 2, 0).detach().cpu().numpy()) at around line 142 in igs2gs_pipeline.py). The pipeline should be working if the saved debug image appears edited.

The pipeline will first edit all images in the training set (which takes about few seconds per image) and then trains for 2.5k, then it repeats this process, so the splat won't appear edited until the full dataset is updated (should be a few min for ~100 image dataset).

Let me know if you run into any other problems

[StephenYangjz]

Hi @cvachha Thanks for your reply. I tried exactly what you suggested before already, but it seems like the igs2gs_pipeline.py is not even up running...

I just pulled gsplat, igs2gs, and nerfstudio and pip install -e .'ed all of them. It seems like tho, the code will have the error message below if I run the same command:

(nerfstudio) stephen@itx:~/Desktop/dreamscenes$ ns-train igs2gs --data conditioning/gen_dataset --load-dir outputs/unnamed/splatfacto/2024-04-15_221736/nerfstudio_models --pipeline.model.prompt "turn the bed into a fire pit" blender-data
Traceback (most recent call last):
  File "/home/stephen/.local/bin/ns-train", line 5, in <module>
    from nerfstudio.scripts.train import entrypoint
  File "/home/stephen/Desktop/dreamscenes/nerfstudio/nerfstudio/scripts/train.py", line 62, in <module>
    from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
  File "/home/stephen/Desktop/dreamscenes/nerfstudio/nerfstudio/configs/method_configs.py", line 728, in <module>
    all_methods, all_descriptions = merge_methods(all_methods, all_descriptions, *discover_methods())
                                                                                  ^^^^^^^^^^^^^^^^^^
  File "/home/stephen/Desktop/dreamscenes/nerfstudio/nerfstudio/plugins/registry.py", line 43, in discover_methods
    spec = discovered_entry_points[name].load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/importlib_metadata/__init__.py", line 203, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stephen/Desktop/dreamscenes/instruct-gs2gs/igs2gs/igs2gs_config.py", line 32, in <module>
    from igs2gs.igs2gs_datamanager import InstructGS2GSDataManagerConfig
  File "/home/stephen/Desktop/dreamscenes/instruct-gs2gs/igs2gs/igs2gs_datamanager.py", line 87, in <module>
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'nerfstudio.data.dataparsers.nerfstudio_dataparser.NerfstudioDataParserConfig'> for field dataparser is not allowed: use default_factory

And it seems like commenting out the @dataclass solves this error but gives me the issue commented above.

[StephenYangjz]

That was Python version 3.11 -- and I tried to install everything in python 3.8 as well and it still gives me the same error. It seems like default_factory is correctly used already in InstructGS2GSDataManagerConfig, so im not sure where the error comes from. @cvachha Any pointers would be greatly appreciated!

[StephenYangjz]

bumping up the message @cvachha thanks

[cvachha]

Hi, I haven't seen this issue before, so I'm not sure what could be causing it. Last case scenario, you can try deleting and re-installing Instruct-GS2GS. I'm wondering if these errors occur when you use other nerfstudio methods or other custom methods.

There was a similar issue on the nerfstudio discord here. One of the proposed solutions mentions trying with python 3.10, but I'm not sure if that will fix it. If using python 3.10 doesn't resolve it, maybe you can try asking on the nerfstudio discord and see if someone else has encountered that error before.

[StephenYangjz]

Thank you! Turns out its still the issue with python version 3.11, I somehow had a global ns installation that runs in python 3.11 that overruns the one installed in the local conda env. Hope that helped others as well : )

[cvachha]

Glad it worked! Thanks for letting me know