search

google-research / sparf

This is the official code release for SPARF: Neural Radiance Fields from Sparse and Noisy Poses [CVPR 2023-Highlight]
https://prunetruong.com/sparf.github.io/
Apache License 2.0
285 stars 15 forks source link

error when during running on llff dataset #9

Closed JunyuanDeng closed 1 year ago

JunyuanDeng commented 1 year ago

Thanks for your great job! I encountered a bug when I test on llff dataset:

(sparf-env) juneyoung@juneyoung:~/Programmes/sparf$ python run_trainval.py joint_pose_nerf_training/llff sparf --scene=orchids --train_sub 3
Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Loading model from: /home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/lpips/weights/v0.1/vgg.pth
/home/juneyoung/Programmes/sparf/source/utils/colmap_initialization/../../../third_party/DenseMatching
Training:  joint_pose_nerf_training/llff/subset_3/orchids  sparf
Date: 27/09/2023
(creating new options file...)
[2023-09-27 19:46:02] [INFO] Command executed: run_trainval.py joint_pose_nerf_training/llff sparf --scene=orchids --train_sub 3
[2023-09-27 19:46:02] [INFO] Training:  joint_pose_nerf_training/llff/subset_3/orchids  sparf
Date: 27/09/2023
[2023-09-27 19:46:02] [INFO] Tensorboard is enabled. Write events to /home/juneyoung/Programmes/sparf/run_cache/joint_pose_nerf_training/llff/subset_3/orchids/sparf.
[2023-09-27 19:46:02] [INFO] Using Single-GPU mode.
[2023-09-27 19:46:02] [INFO] Using device cuda
[2023-09-27 19:46:02] [INFO] Setting seed 0
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Loading model from: /home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/lpips/weights/v0.1/alex.pth
[2023-09-27 19:46:03] [INFO] loading training data...
training dataset: llff
Train dataset llff has 3 samples
number of samples: 3
[2023-09-27 19:46:03] [INFO] loading test data...
eval dataset: llff
eval scenes: orchids
Val dataset llff has 4 samples
number of samples: 4
/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
[2023-09-27 19:46:03] [INFO] Depth type inverse
[2023-09-27 19:46:03] [INFO] depth range 1 to 0
[2023-09-27 19:46:03] [INFO] building networks...
/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Loading flow checkpoint from /home/juneyoung/Programmes/sparf/PDCNet_megadepth.pth.tar
[2023-09-27 19:46:04] [CRIT] Found 3/3 valid initial poses
[2023-09-27 19:46:04] [CRIT] All initial poses: error_R_before_align: 6.202, error_t_before_align: 0.258, error_R: nan, error_t: nan
[2023-09-27 19:46:04] [INFO] Creating NerF model for joint pose-NeRF training
Traceback (most recent call last):
  File "/home/juneyoung/Programmes/sparf/run_trainval.py", line 156, in <module>
    main()
  File "/home/juneyoung/Programmes/sparf/run_trainval.py", line 151, in main
    run_training(args.train_module, args.train_name, cudnn_benchmark=args.cudnn_benchmark, seed=args.seed,
  File "/home/juneyoung/Programmes/sparf/run_trainval.py", line 105, in run_training
    trainer = define_trainer(args=settings, settings_model=model_config, 
  File "/home/juneyoung/Programmes/sparf/source/training/define_trainer.py", line 97, in define_trainer
    trainer = PoseAndNerfTrainerPerScene(args)
  File "/home/juneyoung/Programmes/sparf/source/training/joint_pose_nerf_trainer.py", line 414, in __init__
    super().__init__(opt)
  File "/home/juneyoung/Programmes/sparf/source/training/nerf_trainer.py", line 51, in __init__
    self.init_for_training(opt)
  File "/home/juneyoung/Programmes/sparf/source/training/nerf_trainer.py", line 60, in init_for_training
    self.build_networks(opt) 
  File "/home/juneyoung/Programmes/sparf/source/training/joint_pose_nerf_trainer.py", line 489, in build_networks
    plotting_dict = self.visualize_poses\
  File "/home/juneyoung/Programmes/sparf/source/training/joint_pose_nerf_trainer.py", line 379, in visualize_poses
    return self.visualize_any_poses(opt, pose, pose_ref_, step, split=split)
  File "/home/juneyoung/Programmes/sparf/source/training/joint_pose_nerf_trainer.py", line 338, in visualize_any_poses
    pose_vis = util_vis.plot_save_poses(opt,fig,pose_aligned,pose_ref_w2c=pose_ref,ep=self.iteration)
  File "/home/juneyoung/Programmes/sparf/source/utils/vis_rendering.py", line 362, in plot_save_poses
    ax1.scatter(cam[i,5,0],cam[i,5,1],cam[i,5,2],color=c,s=40)
  File "/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/matplotlib/__init__.py", line 1465, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/home/juneyoung/anaconda3/envs/sparf-env/lib/python3.9/site-packages/mpl_toolkits/mplot3d/axes3d.py", line 2377, in scatter
    if kwargs.get('color', None):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Could you please provide some ideas to solve it?

saurabheights commented 1 year ago

@JunyuanDeng Just passing through. The error

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() happens when you check if a numpy array is true, which doesnt make sense. How can an array be True, one should convert array of bool to bool?

Now, you are getting error here if kwargs.get('color', None):, which means kwargs has color key with a numpy array as value. This is causing the problem. But this code is of matplotlib, not sparf

The sparf code which calls matplotlib is:- ax1.scatter(cam[i,5,0],cam[i,5,1],cam[i,5,2],color=c,s=40) which also looks fine.

Digging deeper into matplotlib and looks like a recent change broke this - https://github.com/matplotlib/matplotlib/issues/26821

Maybe downgrade or upgrade matplotlib.

P.S. @ PruneTruong - Nice work. Following this repo to have fun when I have bigger gpu. :)

JunyuanDeng commented 1 year ago

@saurabheights Thanks for your help! I will try it later!