Autodetected carla dataset
Namespace(gpus=4, dataset='carla', xid='', resolution=128, batch_size=2, run_inversion=True, resume_from='g_carla_pretrained', root_path='.', data_path='../data/nerf', iterations=300000, lr_g=0.0025, lr_d=0.002, dual_discriminator=False, dual_discriminator_l1=False, dual_discriminator_mse=False, r1=5.0, tv=0.5, entropy=0.05, eikonal=0.1, supervise_alpha=False, conditional_pose=True, augment_p=0, augment_ada=False, ada_target=0.6, path_length_regularization=False, perturb_poses=0, clip_gradient_norm=100.0, fine_sampling=True, attention_values=10, use_sdf=True, use_encoder=False, use_viewdir=False, use_class=False, latent_dim=512, disable_stylegan_noise=True, inv_use_testset=False, inv_use_imagenet_testset=False, inv_use_separate=False, inv_loss='vgg', inv_gain_z=5, inv_steps=None, inv_no_split=False, inv_no_optimize_pose=False, inv_train_coord_only=False, inv_encoder_only=False, inv_export_demo_sample=True, inv_manual_input_path=None, coord_resume_from=None)
Experiment name g_carla_res128_bs2_d512_lrg_0.0025_lrd_0.002_r1_5.0_entropy_0.05_tv_0.5_fine_sdf_eik0.1_attn10_noalpha_pose_nonoise
Saving checkpoints to ./gan_checkpoints/g_carla_res128_bs2_d512_lrg_0.0025_lrd_0.002_r1_5.0_entropy_0.05_tv_0.5_fine_sdf_eik0.1_attn10_noalpha_pose_nonoise
Saving tensorboard logs to ./gan_logs/g_carla_res128_bs2_d512_lrg_0.0025_lrd_0.002_r1_5.0_entropy_0.05_tv_0.5_fine_sdf_eik0.1_attn10_noalpha_pose_nonoise
Saving inversion reports to ./reports
Attempting to load latest checkpoint...
Resuming from manual checkpoint ./gan_checkpoints/g_carla_pretrained/checkpoint_latest.pth
Checkpoint iteration: 300000
Loading data...
10000 images
100%|██████████████████████████████████████████████████████████████████████████████████████████| 313/313 [00:36<00:00, 8.62it/s]
torch.Size([10000, 128, 128, 3])
Initializing Inception network, tensorflow weights...
Evaluating training FID on 8000 images
Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
/opt/conda/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/opt/conda/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /opt/conda/lib/python3.10/site-packages/lpips/weights/v0.1/vgg.pth
Params G: 32.189208 M
Params D_0: 28.921088 M
Effective G lr: 0.0025
Effective D lr: 0.002
Loading specified checkpoint...
Resuming GAN from iteration 300000
Config string: i_train_joint_optpose_vgg_gain5_split_it300000
Saving report in ./reports/g_carla_pretrained/i_train_joint_optpose_vgg_gain5_split_it300000
Experiment name c_it300000
Saving to ./coords_checkpoints/g_carla_pretrained/c_it300000_latest.pth
Restoring encoder checkpoint from ./coords_checkpoints/g_carla_pretrained/c_it300000_latest.pth
Initialized SegFormer B5 backbone (84986304 parameters)
Resuming encoder from iteration 120000...
Running...
Traceback (most recent call last):
File "/home/shared/RX0251_wangfeifan/workspace/nerf-from-image-main/run.py", line 1968, in <module>
target_coords, target_mask, target_w = coord_regressor(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
output.reraise()
File "/opt/conda/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
output = module(*input, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/shared/RX0251_wangfeifan/workspace/nerf-from-image-main/models/encoder.py", line 71, in forward
features = self.backbone(x)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/shared/RX0251_wangfeifan/workspace/nerf-from-image-main/models/segformer.py", line 251, in forward
x = blk(x, height, width)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/shared/RX0251_wangfeifan/workspace/nerf-from-image-main/models/segformer.py", line 125, in forward
x = x + self.drop_path(self.attn(self.norm1(x), height, width))
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/shared/RX0251_wangfeifan/workspace/nerf-from-image-main/models/segformer.py", line 95, in forward
x_ = self.sr(x_).reshape(bs, c, -1).permute(0, 2, 1)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, 64, -1] because the unspecified dimension size -1 can be any value and is ambiguous