search

OPEN-AIR-SUN / mars

MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving
Apache License 2.0
681 stars 64 forks source link

Why is the PSNR good during training and bad during testing? #41

Closed jelleopard closed 1 year ago

jelleopard commented 1 year ago

I trained on kitti 0006 dataset with high PSNR but tested with low PSNR, what is the cause of this thanks in advance for your reply!

wuzirui commented 1 year ago

Hi! Can you provide more specific training info?

jelleopard commented 1 year ago

Thanks for your reply. The results as follows:

The image sequence I used is as follows： first_frame: 65 last_frame: 120 The image of scene 0002 is used as testing, and 0002 and 0003 are used as training. cicai_configs.py KITTI_Recon_NSG_Car_Depth = MethodSpecification( config=TrainerConfig( method_name="nsg-kitti-car-depth-recon", steps_per_eval_image=STEPS_PER_EVAL_IMAGE, steps_per_eval_all_images=STEPS_PER_EVAL_ALL_IMAGES, steps_per_save=STEPS_PER_SAVE, max_num_iterations=MAX_NUM_ITERATIONS, save_only_latest_checkpoint=False, mixed_precision=False, use_grad_scaler=True, log_gradients=True, pipeline=NSGPipelineConfig( datamanager=NSGkittiDataManagerConfig( dataparser=NSGkittiDataParserConfig( use_car_latents=True, use_depth=True, use_semantic=False, split_setting="reconstruction", car_object_latents_path=Path( "/home/mars/latents/KITTI-MOT/car-object-latents/latent_codes06.pt" ), car_nerf_state_dict_path=Path("/home/mars/latents/KITTI-MOT/car-nerf-state-dict/epoch_670.ckpt"), ), train_num_rays_per_batch=4096, eval_num_rays_per_batch=4096, camera_optimizer=CameraOptimizerConfig(mode="off"), ), model=SceneGraphModelConfig( mono_depth_loss_mult=0.05, depth_loss_mult=0, background_model=NerfactoModelConfig(),

object_model_template=NerfactoModelConfig(),

            object_model_template=CarNeRFModelConfig(_target=CarNeRF),
            object_representation="class-wise",
            object_ray_sample_strategy="remove-bg",
        ),
    ),
    optimizers={
        "background_model": {
            "optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
            "scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
        },
        "learnable_global": {
            "optimizer": RAdamOptimizerConfig(lr=1e-3, eps=1e-15),
            "scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
        },
        "object_model": {
            "optimizer": RAdamOptimizerConfig(lr=5e-3, eps=1e-15),
            "scheduler": ExponentialDecaySchedulerConfig(lr_final=1e-5, max_steps=200000),
        },
    },
    # viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
    vis="wandb",
),
description="Neural Scene Graph implementation with vanilla-NeRF model for backgruond and object models.",

)

The config.yml as follows:

!!python/object:nerfstudio.engine.trainer.TrainerConfig _target: !!python/name:nerfstudio.engine.trainer.Trainer '' data: &id003 !!python/object/apply:pathlib.PosixPath

wuzirui commented 1 year ago

A gap of 3 in PSNR between training and testing is normal.

AmazingRoad commented 1 year ago

@jelleopard hi, did you use the default config of 'KITTI_NVS_NSG_Car_Depth'? I see your pnsr is about 30, but my model pnsr is only less than 22....