search

NVlabs / neuralangelo

Official implementation of "Neuralangelo: High-Fidelity Neural Surface Reconstruction" (CVPR 2023)
https://research.nvidia.com/labs/dir/neuralangelo/
Other
4.27k stars 383 forks source link

KeyError: 'sk_x' when using fox data from instant-ngp #183

Open alexwang2013 opened 5 months ago

alexwang2013 commented 5 months ago

I want check the result by using the fox data from which is here: fox

I executed the following command:

EXPERIMENT=fox
GROUP=fox
NAME=fox
CONFIG=projects/neuralangelo/configs/custom/${EXPERIMENT}.yaml
GPUS=1  # use >1 for multi-GPU training!
torchrun --nproc_per_node=${GPUS} train.py \
    --logdir=logs/${GROUP}/${NAME} \
    --config=${CONFIG} \
    --show_pbar

output is the following

(Setting affinity with NVML failed, skipping...)
[W Utils.hpp:133] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarInt)
Training with 1 GPUs.
Using random seed 0
Make folder logs/fox/fox
* checkpoint:
   * save_epoch: 9999999999
   * save_iter: 20000
   * save_latest_iter: 9999999999
   * save_period: 9999999999
   * strict_resume: True
* cudnn:
   * benchmark: True
   * deterministic: False
* data:
   * name: dummy
   * num_images: None
   * num_workers: 4
   * preload: True
   * readjust:
      * center: [0.0, 0.0, 0.0]
      * scale: 1.0
   * root: /mnt/d/wang_back_data/neuralangelo/fox
   * train:
      * batch_size: 2
      * image_size: [1920, 1080]
      * subset: None
   * type: projects.neuralangelo.data
   * use_multi_epoch_loader: True
   * val:
      * batch_size: 2
      * image_size: [533, 300]
      * max_viz_samples: 16
      * subset: 4
* image_save_iter: 9999999999
* inference_args:
* local_rank: 0
* logdir: logs/fox/fox
* logging_iter: 9999999999999
* max_epoch: 9999999999
* max_iter: 500000
* metrics_epoch: None
* metrics_iter: None
* model:
   * appear_embed:
      * dim: 8
      * enabled: False
   * background:
      * enabled: True
      * encoding:
         * levels: 10
         * type: fourier
      * encoding_view:
         * levels: 3
         * type: spherical
      * mlp:
         * activ: relu
         * activ_density: softplus
         * activ_density_params:
         * activ_params:
         * hidden_dim: 256
         * hidden_dim_rgb: 128
         * num_layers: 8
         * num_layers_rgb: 2
         * skip: [4]
         * skip_rgb: []
      * view_dep: True
      * white: False
   * object:
      * rgb:
         * encoding_view:
            * levels: 3
            * type: spherical
         * mlp:
            * activ: relu_
            * activ_params:
            * hidden_dim: 256
            * num_layers: 4
            * skip: []
            * weight_norm: True
         * mode: idr
      * s_var:
         * anneal_end: 0.1
         * init_val: 3.0
      * sdf:
         * encoding:
            * coarse2fine:
               * enabled: True
               * init_active_level: 4
               * step: 5000
            * hashgrid:
               * dict_size: 22
               * dim: 8
               * max_logres: 11
               * min_logres: 5
               * range: [-2, 2]
            * levels: 16
            * type: hashgrid
         * gradient:
            * mode: numerical
            * taps: 4
         * mlp:
            * activ: softplus
            * activ_params:
               * beta: 100
            * geometric_init: True
            * hidden_dim: 256
            * inside_out: False
            * num_layers: 1
            * out_bias: 0.5
            * skip: []
            * weight_norm: True
   * render:
      * num_sample_hierarchy: 4
      * num_samples:
         * background: 32
         * coarse: 64
         * fine: 16
      * rand_rays: 512
      * stratified: True
   * type: projects.neuralangelo.model
* nvtx_profile: False
* optim:
   * fused_opt: False
   * params:
      * lr: 0.001
      * weight_decay: 0.01
   * sched:
      * gamma: 10.0
      * iteration_mode: True
      * step_size: 9999999999
      * two_steps: [300000, 400000]
      * type: two_steps_with_warmup
      * warm_up_end: 5000
   * type: AdamW
* pretrained_weight: None
* source_filename: projects/neuralangelo/configs/custom/fox.yaml
* speed_benchmark: False
* test_data:
   * name: dummy
   * num_workers: 0
   * test:
      * batch_size: 1
      * is_lmdb: False
      * roots: None
   * type: imaginaire.datasets.images
* timeout_period: 9999999
* trainer:
   * amp_config:
      * backoff_factor: 0.5
      * enabled: False
      * growth_factor: 2.0
      * growth_interval: 2000
      * init_scale: 65536.0
   * ddp_config:
      * find_unused_parameters: False
      * static_graph: True
   * depth_vis_scale: 0.5
   * ema_config:
      * beta: 0.9999
      * enabled: False
      * load_ema_checkpoint: False
      * start_iteration: 0
   * grad_accum_iter: 1
   * image_to_tensorboard: False
   * init:
      * gain: None
      * type: none
   * loss_weight:
      * curvature: 0.0005
      * eikonal: 0.1
      * render: 1.0
   * type: projects.neuralangelo.trainer
* validation_iter: 5000
* wandb_image_iter: 10000
* wandb_scalar_iter: 100
cudnn benchmark: True
cudnn deterministic: False
Setup trainer.
Using random seed 0
/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
model parameter count: 366,702,732
Initialize model weights using type: none, gain: None
Using random seed 0
[rank0]:[W Utils.hpp:106] Warning: Environment variable NCCL_ASYNC_ERROR_HANDLING is deprecated; use TORCH_NCCL_ASYNC_ERROR_HANDLING instead (function getCvarString)
Allow TensorFloat32 operations on supported devices
preloading cameras (train):   0%|                                                                                                                                                                                           | 0/50 [00:00<?, ?it/s]Exception in thread Thread-6Exception in thread Thread-7Exception in thread :
Thread-8Exception in thread Traceback (most recent call last):
Thread-9:
Traceback (most recent call last):
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
:
:
Traceback (most recent call last):
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Traceback (most recent call last):
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self.run()
  File "/mnt/d/wang/linux/tools/miniconda3/envs/neuralangelo/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    self._target(*self._args, **self._kwargs)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/nerf/datasets/base.py", line 29, in _preload_worker
    data_list[idx] = load_func(idx)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
    data_list[idx] = load_func(idx)
      File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
data_list[idx] = load_func(idx)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
    data_list[idx] = load_func(idx)
  File "/mnt/c/Users/1/wang/3D-AI/neuralangelo/projects/neuralangelo/data.py", line 103, in get_camera
    intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],
intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],KeyError
: 'sk_x'KeyError
: 'sk_x'
    intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],
KeyError: 'sk_x'
    intr = torch.tensor([[self.meta["fl_x"], self.meta["sk_x"], self.meta["cx"]],
KeyError: 'sk_x'

Environment

By the way for the data fox there are some frames in the transform.json file, but there is no relative jpg file in the folder images. this will cause exceptions, so I removed the unused frames from transform.json. For e.g. image 0016.jpg is not in folder images, I removed the following frame from transform.json.

{
      "file_path": "images/0016.jpg",
      "sharpness": 15.184317835374785,
      "transform_matrix": [
        [
          0.34747494861304573,
          0.053380418010164174,
          0.9361685961577728,
          5.639885048014769
        ],
        [
          0.9356011023082821,
          0.04685455607320249,
          -0.3499359634577667,
          -2.679224620930034
        ],
        [
          -0.06254348426497747,
          0.9974744008232991,
          -0.033661993185310046,
          -0.6582774295652742
        ],
        [
          0.0,
          0.0,
          0.0,
          1.0
        ]
      ]
    },
Bombaninha commented 4 months ago

The transforms.json file format for Instant-NGP and Neuralangelo differs in some aspects. Apparently, your file does not have information about skew. Ideally, you should go through the preprocessing step of Neuralangelo to update this file in the correct format. This can be seen in: neuralangelo/projects/neuralangelo/scripts/convert_data_to_json.py