KeyError: 'proposal_networks'

[CaffeyChen]

Traceback (most recent call last):
  File "/home/ycf/anaconda3/envs/nerfstudio/bin/ns-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/ycf/cjh/nerfstudio/nerfstudio/scripts/train.py", line 261, in entrypoint
    main(
  File "/home/ycf/cjh/nerfstudio/nerfstudio/scripts/train.py", line 246, in main
    launch(
  File "/home/ycf/cjh/nerfstudio/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/ycf/cjh/nerfstudio/nerfstudio/scripts/train.py", line 99, in train_loop
    trainer.setup()
  File "/home/ycf/cjh/nerfstudio/nerfstudio/engine/trainer.py", line 185, in setup
    self._load_checkpoint()
  File "/home/ycf/cjh/nerfstudio/nerfstudio/engine/trainer.py", line 411, in _load_checkpoint
    self.optimizers.load_schedulers(loaded_state["schedulers"])
  File "/home/ycf/cjh/nerfstudio/nerfstudio/engine/optimizers.py", line 170, in load_schedulers
    self.schedulers[k].load_state_dict(v)
KeyError: 'proposal_networks'

I use the official dataset 'bear'. I trained the initial NeRF with the following command: ns-train nerfacto --data {bear} To start training for editing the NeRF, run the following command: ns-train in2n --data bear --load-dir outputs/bear/.../nerfstudio_models --pipeline.prompt {"panda bear"} --pipeline.guidance-scale 7.5 --pipeline.image-guidance-scale 1.5 And finally got 'KeyError: proposal_networks'.

Is there any method to solve this problem?

[ayaanzhaque]

Can you replace your argument to --load-dir with the actual path to the bear checkpoint? I think that is your error. Also, remove the braces from the prompt and see if that fixes anything.

[CaffeyChen]

Hi, Thank you for your reply. My full command is the following: ns-train in2n --data data_in2n/bear --load-dir outputs/bear/nerfacto/2023-08-07_193530/nerfstudio_models --pipeline.prompt "panda bear" But I got the same error. I found in optimizers.py, the self.schedulers only has Key camera_optand doesn't have proposal_networks and fields.

However, load_state["schedulers"] content is 'camera_opt': {'base_lrs': [0.0006], 'last_epoch': 29985, 'verbose': False, '_step_count': 29986, '_get_lr_called_within_step': False, '_last_lr': [0.00030081622047758825], 'lr_lambdas': [None]}, 'proposal_networks': {'base_lrs': [0.01], 'last_epoch': 29985, 'verbose': False, '_step_count': 29986, '_get_lr_called_within_step': False, '_last_lr': [0.0050136036746264755], 'lr_lambdas': [None]}, 'fields': {'base_lrs': [0.01], 'last_epoch': 29985, 'verbose': False, '_step_count': 29986, '_get_lr_called_within_step': False, '_last_lr': [0.0050136036746264755], 'lr_lambdas': [None]}}

I thought this is the reason why I got the error. Is there any way to fix it? Or are there any other errors?

[ayaanzhaque]

I see, there must have been a change to the main nerfacto config. Can you edit the in2n_config to match the same parameters for camera opt as the current public nerfstudio?

[CaffeyChen]

Well, cause I am a nerfstudio beginner, I am not good at searching the correct scripts and codes. Can you tell me which in2n_config should be editted? Thank you!

[ayaanzhaque]

https://github.com/ayaanzhaque/instruct-nerf2nerf/blob/main/in2n/in2n_config.py#L30-L69

Can you replace this section of code with

in2n_method = MethodSpecification(
    config=InstructNeRF2NeRFTrainerConfig(
        method_name="in2n",
        steps_per_eval_batch=1000,
        steps_per_eval_image=100,
        steps_per_save=250,
        max_num_iterations=15000,
        save_only_latest_checkpoint=True,
        mixed_precision=True,
        pipeline=InstructNeRF2NeRFPipelineConfig(
            datamanager=InstructNeRF2NeRFDataManagerConfig(
                dataparser=NerfstudioDataParserConfig(),
                train_num_rays_per_batch=16384,
                eval_num_rays_per_batch=4096,
                patch_size=32,
                camera_optimizer=CameraOptimizerConfig(
                    mode="SO3xR3",
                    optimizer=AdamOptimizerConfig(lr=6e-4, eps=1e-8, weight_decay=1e-2),
                    scheduler=ExponentialDecaySchedulerConfig(lr_final=6e-6, max_steps=200000),
                ),
            ),
            model=InstructNeRF2NeRFModelConfig(
                eval_num_rays_per_chunk=1 << 15,
                use_lpips=True,
            ),
            ip2p_use_full_precision=True
        ),
        optimizers={
            "proposal_networks": {
                "optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15),
                "scheduler": ExponentialDecaySchedulerConfig(lr_final=0.0001, max_steps=200000),
            },
            "fields": {
                "optimizer": AdamOptimizerConfig(lr=1e-2, eps=1e-15),
                "scheduler": ExponentialDecaySchedulerConfig(lr_final=0.0001, max_steps=200000),
            },
        },
        viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
        vis="viewer",
    ),
    description="Instruct-NeRF2NeRF primary method: uses LPIPS, IP2P at full precision",
)

[Boyyeo]

Thanks, it helps ! and you will need to include this line

from nerfstudio.engine.schedulers import ExponentialDecaySchedulerConfig

Furthermore, could you please inform me of the time required to edit a scene?

[ayaanzhaque]

Great! If either of you wants to make a PR to update the config, that would be great