search

Lakonik / SSDNeRF

[ICCV 2023] Single-Stage Diffusion NeRF
https://lakonik.github.io/ssdnerf/
MIT License
430 stars 23 forks source link

Test using the provided weights on custom data #30

Open XuM007 opened 10 months ago

XuM007 commented 10 months ago

Thank you very much for your work. I now want to use the model(ssdnerf_chairs_recons1v_80k_emaonly.pth) you provided to reconstruct my own data. I reconstructed a mesh from the chair image I provided below, but it seems to have some discrepancies with reality. 3_static_crop_resize image

Therefore, I would like to test with two or four images. I have a few questions:

  1. Do I need to use the training set to train the model weights for recons2v/recons4v, or can I use the weights for recons1v?
  2. For my data, there is some error in the camera poses for multiple views, and they are not very accurate. However, precise camera poses are crucial for NeRF. I'm not sure if this will affect the performance of SSDNeRF.
  3. Regarding camera intrinsic parameters, I've noticed that my camera has distortion parameters. Additionally, to achieve a (128, 128) image size, I cropped and resized my original images, which introduced some errors in the calculation of the camera intrinsic parameters. I would like to know if such camera intrinsic parameters will affect the model's performance.
  4. When I used the downloaded weights and added my own data to the test, I obtained the following output:

2023-11-08 13:30:22,626 - mmgen - INFO - evaluation 2023-11-08 13:30:22,626 - mmgen - INFO - Set random seed to 2021, deterministic: False, use_rank_shift: False 2023-11-08 13:30:23,048 - mmgen - INFO - Apply 'timestep_weight' rescale_mode for loss_ddpm_mse. Please make sure the passed weight can be updated by external functions. load checkpoint from local path: work_dirs/cache/ssdnerf_chairs_recons1v_80k_emaonly.pth The model and loaded state dict do not match exactly missing keys in source state_dict: decoder.aabb, decoder.base_net.0.weight, decoder.base_net.0.bias, decoder.density_net.0.weight, decoder.density_net.0.bias, decoder.dir_net.0.weight, decoder.dir_net.0.bias, decoder.color_net.0.weight, decoder.color_net.0.bias, diffusion.denoising.time_embedding.blocks.0.weight, diffusion.denoising.time_embedding.blocks.0.bias, diffusion.denoising.time_embedding.blocks.2.weight, diffusion.denoising.time_embedding.blocks.2.bias, diffusion.denoising.in_blocks.0.0.weight, diffusion.denoising.in_blocks.0.0.bias, diffusion.denoising.in_blocks.1.0.conv_1.0.weight, diffusion.denoising.in_blocks.1.0.conv_1.0.bias, diffusion.denoising.in_blocks.1.0.conv_1.2.weight, diffusion.denoising.in_blocks.1.0.conv_1.2.bias, diffusion.denoising.in_blocks.1.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.1.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.1.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.1.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.1.0.conv_2.2.weight, diffusion.denoising.in_blocks.1.0.conv_2.2.bias, diffusion.denoising.in_blocks.2.0.conv_1.0.weight, diffusion.denoising.in_blocks.2.0.conv_1.0.bias, diffusion.denoising.in_blocks.2.0.conv_1.2.weight, diffusion.denoising.in_blocks.2.0.conv_1.2.bias, diffusion.denoising.in_blocks.2.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.2.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.2.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.2.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.2.0.conv_2.2.weight, diffusion.denoising.in_blocks.2.0.conv_2.2.bias, diffusion.denoising.in_blocks.3.0.downsample.weight, diffusion.denoising.in_blocks.3.0.downsample.bias, diffusion.denoising.in_blocks.4.0.conv_1.0.weight, diffusion.denoising.in_blocks.4.0.conv_1.0.bias, diffusion.denoising.in_blocks.4.0.conv_1.2.weight, diffusion.denoising.in_blocks.4.0.conv_1.2.bias, diffusion.denoising.in_blocks.4.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.4.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.4.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.4.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.4.0.conv_2.2.weight, diffusion.denoising.in_blocks.4.0.conv_2.2.bias, diffusion.denoising.in_blocks.4.0.shortcut.weight, diffusion.denoising.in_blocks.4.0.shortcut.bias, diffusion.denoising.in_blocks.5.0.conv_1.0.weight, diffusion.denoising.in_blocks.5.0.conv_1.0.bias, diffusion.denoising.in_blocks.5.0.conv_1.2.weight, diffusion.denoising.in_blocks.5.0.conv_1.2.bias, diffusion.denoising.in_blocks.5.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.5.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.5.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.5.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.5.0.conv_2.2.weight, diffusion.denoising.in_blocks.5.0.conv_2.2.bias, diffusion.denoising.in_blocks.6.0.downsample.weight, diffusion.denoising.in_blocks.6.0.downsample.bias, diffusion.denoising.in_blocks.7.0.conv_1.0.weight, diffusion.denoising.in_blocks.7.0.conv_1.0.bias, diffusion.denoising.in_blocks.7.0.conv_1.2.weight, diffusion.denoising.in_blocks.7.0.conv_1.2.bias, diffusion.denoising.in_blocks.7.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.7.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.7.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.7.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.7.0.conv_2.2.weight, diffusion.denoising.in_blocks.7.0.conv_2.2.bias, diffusion.denoising.in_blocks.7.1.norm.weight, diffusion.denoising.in_blocks.7.1.norm.bias, diffusion.denoising.in_blocks.7.1.qkv.weight, diffusion.denoising.in_blocks.7.1.qkv.bias, diffusion.denoising.in_blocks.7.1.proj.weight, diffusion.denoising.in_blocks.7.1.proj.bias, diffusion.denoising.in_blocks.8.0.conv_1.0.weight, diffusion.denoising.in_blocks.8.0.conv_1.0.bias, diffusion.denoising.in_blocks.8.0.conv_1.2.weight, diffusion.denoising.in_blocks.8.0.conv_1.2.bias, diffusion.denoising.in_blocks.8.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.8.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.8.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.8.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.8.0.conv_2.2.weight, diffusion.denoising.in_blocks.8.0.conv_2.2.bias, diffusion.denoising.in_blocks.8.1.norm.weight, diffusion.denoising.in_blocks.8.1.norm.bias, diffusion.denoising.in_blocks.8.1.qkv.weight, diffusion.denoising.in_blocks.8.1.qkv.bias, diffusion.denoising.in_blocks.8.1.proj.weight, diffusion.denoising.in_blocks.8.1.proj.bias, diffusion.denoising.in_blocks.9.0.downsample.weight, diffusion.denoising.in_blocks.9.0.downsample.bias, diffusion.denoising.in_blocks.10.0.conv_1.0.weight, diffusion.denoising.in_blocks.10.0.conv_1.0.bias, diffusion.denoising.in_blocks.10.0.conv_1.2.weight, diffusion.denoising.in_blocks.10.0.conv_1.2.bias, diffusion.denoising.in_blocks.10.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.10.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.10.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.10.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.10.0.conv_2.2.weight, diffusion.denoising.in_blocks.10.0.conv_2.2.bias, diffusion.denoising.in_blocks.10.0.shortcut.weight, diffusion.denoising.in_blocks.10.0.shortcut.bias, diffusion.denoising.in_blocks.10.1.norm.weight, diffusion.denoising.in_blocks.10.1.norm.bias, diffusion.denoising.in_blocks.10.1.qkv.weight, diffusion.denoising.in_blocks.10.1.qkv.bias, diffusion.denoising.in_blocks.10.1.proj.weight, diffusion.denoising.in_blocks.10.1.proj.bias, diffusion.denoising.in_blocks.11.0.conv_1.0.weight, diffusion.denoising.in_blocks.11.0.conv_1.0.bias, diffusion.denoising.in_blocks.11.0.conv_1.2.weight, diffusion.denoising.in_blocks.11.0.conv_1.2.bias, diffusion.denoising.in_blocks.11.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.11.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.11.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.11.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.11.0.conv_2.2.weight, diffusion.denoising.in_blocks.11.0.conv_2.2.bias, diffusion.denoising.in_blocks.11.1.norm.weight, diffusion.denoising.in_blocks.11.1.norm.bias, diffusion.denoising.in_blocks.11.1.qkv.weight, diffusion.denoising.in_blocks.11.1.qkv.bias, diffusion.denoising.in_blocks.11.1.proj.weight, diffusion.denoising.in_blocks.11.1.proj.bias, diffusion.denoising.in_blocks.12.0.downsample.weight, diffusion.denoising.in_blocks.12.0.downsample.bias, diffusion.denoising.in_blocks.13.0.conv_1.0.weight, diffusion.denoising.in_blocks.13.0.conv_1.0.bias, diffusion.denoising.in_blocks.13.0.conv_1.2.weight, diffusion.denoising.in_blocks.13.0.conv_1.2.bias, diffusion.denoising.in_blocks.13.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.13.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.13.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.13.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.13.0.conv_2.2.weight, diffusion.denoising.in_blocks.13.0.conv_2.2.bias, diffusion.denoising.in_blocks.13.1.norm.weight, diffusion.denoising.in_blocks.13.1.norm.bias, diffusion.denoising.in_blocks.13.1.qkv.weight, diffusion.denoising.in_blocks.13.1.qkv.bias, diffusion.denoising.in_blocks.13.1.proj.weight, diffusion.denoising.in_blocks.13.1.proj.bias, diffusion.denoising.in_blocks.14.0.conv_1.0.weight, diffusion.denoising.in_blocks.14.0.conv_1.0.bias, diffusion.denoising.in_blocks.14.0.conv_1.2.weight, diffusion.denoising.in_blocks.14.0.conv_1.2.bias, diffusion.denoising.in_blocks.14.0.norm_with_embedding.norm.weight, diffusion.denoising.in_blocks.14.0.norm_with_embedding.norm.bias, diffusion.denoising.in_blocks.14.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.in_blocks.14.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.in_blocks.14.0.conv_2.2.weight, diffusion.denoising.in_blocks.14.0.conv_2.2.bias, diffusion.denoising.in_blocks.14.1.norm.weight, diffusion.denoising.in_blocks.14.1.norm.bias, diffusion.denoising.in_blocks.14.1.qkv.weight, diffusion.denoising.in_blocks.14.1.qkv.bias, diffusion.denoising.in_blocks.14.1.proj.weight, diffusion.denoising.in_blocks.14.1.proj.bias, diffusion.denoising.mid_blocks.0.conv_1.0.weight, diffusion.denoising.mid_blocks.0.conv_1.0.bias, diffusion.denoising.mid_blocks.0.conv_1.2.weight, diffusion.denoising.mid_blocks.0.conv_1.2.bias, diffusion.denoising.mid_blocks.0.norm_with_embedding.norm.weight, diffusion.denoising.mid_blocks.0.norm_with_embedding.norm.bias, diffusion.denoising.mid_blocks.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.mid_blocks.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.mid_blocks.0.conv_2.2.weight, diffusion.denoising.mid_blocks.0.conv_2.2.bias, diffusion.denoising.mid_blocks.1.norm.weight, diffusion.denoising.mid_blocks.1.norm.bias, diffusion.denoising.mid_blocks.1.qkv.weight, diffusion.denoising.mid_blocks.1.qkv.bias, diffusion.denoising.mid_blocks.1.proj.weight, diffusion.denoising.mid_blocks.1.proj.bias, diffusion.denoising.mid_blocks.2.conv_1.0.weight, diffusion.denoising.mid_blocks.2.conv_1.0.bias, diffusion.denoising.mid_blocks.2.conv_1.2.weight, diffusion.denoising.mid_blocks.2.conv_1.2.bias, diffusion.denoising.mid_blocks.2.norm_with_embedding.norm.weight, diffusion.denoising.mid_blocks.2.norm_with_embedding.norm.bias, diffusion.denoising.mid_blocks.2.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.mid_blocks.2.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.mid_blocks.2.conv_2.2.weight, diffusion.denoising.mid_blocks.2.conv_2.2.bias, diffusion.denoising.out_blocks.0.0.conv_1.0.weight, diffusion.denoising.out_blocks.0.0.conv_1.0.bias, diffusion.denoising.out_blocks.0.0.conv_1.2.weight, diffusion.denoising.out_blocks.0.0.conv_1.2.bias, diffusion.denoising.out_blocks.0.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.0.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.0.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.0.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.0.0.conv_2.2.weight, diffusion.denoising.out_blocks.0.0.conv_2.2.bias, diffusion.denoising.out_blocks.0.0.shortcut.weight, diffusion.denoising.out_blocks.0.0.shortcut.bias, diffusion.denoising.out_blocks.0.1.norm.weight, diffusion.denoising.out_blocks.0.1.norm.bias, diffusion.denoising.out_blocks.0.1.qkv.weight, diffusion.denoising.out_blocks.0.1.qkv.bias, diffusion.denoising.out_blocks.0.1.proj.weight, diffusion.denoising.out_blocks.0.1.proj.bias, diffusion.denoising.out_blocks.1.0.conv_1.0.weight, diffusion.denoising.out_blocks.1.0.conv_1.0.bias, diffusion.denoising.out_blocks.1.0.conv_1.2.weight, diffusion.denoising.out_blocks.1.0.conv_1.2.bias, diffusion.denoising.out_blocks.1.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.1.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.1.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.1.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.1.0.conv_2.2.weight, diffusion.denoising.out_blocks.1.0.conv_2.2.bias, diffusion.denoising.out_blocks.1.0.shortcut.weight, diffusion.denoising.out_blocks.1.0.shortcut.bias, diffusion.denoising.out_blocks.1.1.norm.weight, diffusion.denoising.out_blocks.1.1.norm.bias, diffusion.denoising.out_blocks.1.1.qkv.weight, diffusion.denoising.out_blocks.1.1.qkv.bias, diffusion.denoising.out_blocks.1.1.proj.weight, diffusion.denoising.out_blocks.1.1.proj.bias, diffusion.denoising.out_blocks.2.0.conv_1.0.weight, diffusion.denoising.out_blocks.2.0.conv_1.0.bias, diffusion.denoising.out_blocks.2.0.conv_1.2.weight, diffusion.denoising.out_blocks.2.0.conv_1.2.bias, diffusion.denoising.out_blocks.2.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.2.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.2.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.2.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.2.0.conv_2.2.weight, diffusion.denoising.out_blocks.2.0.conv_2.2.bias, diffusion.denoising.out_blocks.2.0.shortcut.weight, diffusion.denoising.out_blocks.2.0.shortcut.bias, diffusion.denoising.out_blocks.2.1.norm.weight, diffusion.denoising.out_blocks.2.1.norm.bias, diffusion.denoising.out_blocks.2.1.qkv.weight, diffusion.denoising.out_blocks.2.1.qkv.bias, diffusion.denoising.out_blocks.2.1.proj.weight, diffusion.denoising.out_blocks.2.1.proj.bias, diffusion.denoising.out_blocks.2.2.conv.weight, diffusion.denoising.out_blocks.2.2.conv.bias, diffusion.denoising.out_blocks.3.0.conv_1.0.weight, diffusion.denoising.out_blocks.3.0.conv_1.0.bias, diffusion.denoising.out_blocks.3.0.conv_1.2.weight, diffusion.denoising.out_blocks.3.0.conv_1.2.bias, diffusion.denoising.out_blocks.3.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.3.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.3.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.3.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.3.0.conv_2.2.weight, diffusion.denoising.out_blocks.3.0.conv_2.2.bias, diffusion.denoising.out_blocks.3.0.shortcut.weight, diffusion.denoising.out_blocks.3.0.shortcut.bias, diffusion.denoising.out_blocks.3.1.norm.weight, diffusion.denoising.out_blocks.3.1.norm.bias, diffusion.denoising.out_blocks.3.1.qkv.weight, diffusion.denoising.out_blocks.3.1.qkv.bias, diffusion.denoising.out_blocks.3.1.proj.weight, diffusion.denoising.out_blocks.3.1.proj.bias, diffusion.denoising.out_blocks.4.0.conv_1.0.weight, diffusion.denoising.out_blocks.4.0.conv_1.0.bias, diffusion.denoising.out_blocks.4.0.conv_1.2.weight, diffusion.denoising.out_blocks.4.0.conv_1.2.bias, diffusion.denoising.out_blocks.4.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.4.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.4.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.4.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.4.0.conv_2.2.weight, diffusion.denoising.out_blocks.4.0.conv_2.2.bias, diffusion.denoising.out_blocks.4.0.shortcut.weight, diffusion.denoising.out_blocks.4.0.shortcut.bias, diffusion.denoising.out_blocks.4.1.norm.weight, diffusion.denoising.out_blocks.4.1.norm.bias, diffusion.denoising.out_blocks.4.1.qkv.weight, diffusion.denoising.out_blocks.4.1.qkv.bias, diffusion.denoising.out_blocks.4.1.proj.weight, diffusion.denoising.out_blocks.4.1.proj.bias, diffusion.denoising.out_blocks.5.0.conv_1.0.weight, diffusion.denoising.out_blocks.5.0.conv_1.0.bias, diffusion.denoising.out_blocks.5.0.conv_1.2.weight, diffusion.denoising.out_blocks.5.0.conv_1.2.bias, diffusion.denoising.out_blocks.5.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.5.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.5.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.5.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.5.0.conv_2.2.weight, diffusion.denoising.out_blocks.5.0.conv_2.2.bias, diffusion.denoising.out_blocks.5.0.shortcut.weight, diffusion.denoising.out_blocks.5.0.shortcut.bias, diffusion.denoising.out_blocks.5.1.norm.weight, diffusion.denoising.out_blocks.5.1.norm.bias, diffusion.denoising.out_blocks.5.1.qkv.weight, diffusion.denoising.out_blocks.5.1.qkv.bias, diffusion.denoising.out_blocks.5.1.proj.weight, diffusion.denoising.out_blocks.5.1.proj.bias, diffusion.denoising.out_blocks.5.2.conv.weight, diffusion.denoising.out_blocks.5.2.conv.bias, diffusion.denoising.out_blocks.6.0.conv_1.0.weight, diffusion.denoising.out_blocks.6.0.conv_1.0.bias, diffusion.denoising.out_blocks.6.0.conv_1.2.weight, diffusion.denoising.out_blocks.6.0.conv_1.2.bias, diffusion.denoising.out_blocks.6.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.6.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.6.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.6.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.6.0.conv_2.2.weight, diffusion.denoising.out_blocks.6.0.conv_2.2.bias, diffusion.denoising.out_blocks.6.0.shortcut.weight, diffusion.denoising.out_blocks.6.0.shortcut.bias, diffusion.denoising.out_blocks.6.1.norm.weight, diffusion.denoising.out_blocks.6.1.norm.bias, diffusion.denoising.out_blocks.6.1.qkv.weight, diffusion.denoising.out_blocks.6.1.qkv.bias, diffusion.denoising.out_blocks.6.1.proj.weight, diffusion.denoising.out_blocks.6.1.proj.bias, diffusion.denoising.out_blocks.7.0.conv_1.0.weight, diffusion.denoising.out_blocks.7.0.conv_1.0.bias, diffusion.denoising.out_blocks.7.0.conv_1.2.weight, diffusion.denoising.out_blocks.7.0.conv_1.2.bias, diffusion.denoising.out_blocks.7.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.7.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.7.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.7.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.7.0.conv_2.2.weight, diffusion.denoising.out_blocks.7.0.conv_2.2.bias, diffusion.denoising.out_blocks.7.0.shortcut.weight, diffusion.denoising.out_blocks.7.0.shortcut.bias, diffusion.denoising.out_blocks.7.1.norm.weight, diffusion.denoising.out_blocks.7.1.norm.bias, diffusion.denoising.out_blocks.7.1.qkv.weight, diffusion.denoising.out_blocks.7.1.qkv.bias, diffusion.denoising.out_blocks.7.1.proj.weight, diffusion.denoising.out_blocks.7.1.proj.bias, diffusion.denoising.out_blocks.8.0.conv_1.0.weight, diffusion.denoising.out_blocks.8.0.conv_1.0.bias, diffusion.denoising.out_blocks.8.0.conv_1.2.weight, diffusion.denoising.out_blocks.8.0.conv_1.2.bias, diffusion.denoising.out_blocks.8.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.8.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.8.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.8.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.8.0.conv_2.2.weight, diffusion.denoising.out_blocks.8.0.conv_2.2.bias, diffusion.denoising.out_blocks.8.0.shortcut.weight, diffusion.denoising.out_blocks.8.0.shortcut.bias, diffusion.denoising.out_blocks.8.1.norm.weight, diffusion.denoising.out_blocks.8.1.norm.bias, diffusion.denoising.out_blocks.8.1.qkv.weight, diffusion.denoising.out_blocks.8.1.qkv.bias, diffusion.denoising.out_blocks.8.1.proj.weight, diffusion.denoising.out_blocks.8.1.proj.bias, diffusion.denoising.out_blocks.8.2.conv.weight, diffusion.denoising.out_blocks.8.2.conv.bias, diffusion.denoising.out_blocks.9.0.conv_1.0.weight, diffusion.denoising.out_blocks.9.0.conv_1.0.bias, diffusion.denoising.out_blocks.9.0.conv_1.2.weight, diffusion.denoising.out_blocks.9.0.conv_1.2.bias, diffusion.denoising.out_blocks.9.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.9.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.9.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.9.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.9.0.conv_2.2.weight, diffusion.denoising.out_blocks.9.0.conv_2.2.bias, diffusion.denoising.out_blocks.9.0.shortcut.weight, diffusion.denoising.out_blocks.9.0.shortcut.bias, diffusion.denoising.out_blocks.10.0.conv_1.0.weight, diffusion.denoising.out_blocks.10.0.conv_1.0.bias, diffusion.denoising.out_blocks.10.0.conv_1.2.weight, diffusion.denoising.out_blocks.10.0.conv_1.2.bias, diffusion.denoising.out_blocks.10.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.10.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.10.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.10.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.10.0.conv_2.2.weight, diffusion.denoising.out_blocks.10.0.conv_2.2.bias, diffusion.denoising.out_blocks.10.0.shortcut.weight, diffusion.denoising.out_blocks.10.0.shortcut.bias, diffusion.denoising.out_blocks.11.0.conv_1.0.weight, diffusion.denoising.out_blocks.11.0.conv_1.0.bias, diffusion.denoising.out_blocks.11.0.conv_1.2.weight, diffusion.denoising.out_blocks.11.0.conv_1.2.bias, diffusion.denoising.out_blocks.11.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.11.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.11.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.11.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.11.0.conv_2.2.weight, diffusion.denoising.out_blocks.11.0.conv_2.2.bias, diffusion.denoising.out_blocks.11.0.shortcut.weight, diffusion.denoising.out_blocks.11.0.shortcut.bias, diffusion.denoising.out_blocks.11.1.conv.weight, diffusion.denoising.out_blocks.11.1.conv.bias, diffusion.denoising.out_blocks.12.0.conv_1.0.weight, diffusion.denoising.out_blocks.12.0.conv_1.0.bias, diffusion.denoising.out_blocks.12.0.conv_1.2.weight, diffusion.denoising.out_blocks.12.0.conv_1.2.bias, diffusion.denoising.out_blocks.12.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.12.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.12.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.12.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.12.0.conv_2.2.weight, diffusion.denoising.out_blocks.12.0.conv_2.2.bias, diffusion.denoising.out_blocks.12.0.shortcut.weight, diffusion.denoising.out_blocks.12.0.shortcut.bias, diffusion.denoising.out_blocks.13.0.conv_1.0.weight, diffusion.denoising.out_blocks.13.0.conv_1.0.bias, diffusion.denoising.out_blocks.13.0.conv_1.2.weight, diffusion.denoising.out_blocks.13.0.conv_1.2.bias, diffusion.denoising.out_blocks.13.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.13.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.13.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.13.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.13.0.conv_2.2.weight, diffusion.denoising.out_blocks.13.0.conv_2.2.bias, diffusion.denoising.out_blocks.13.0.shortcut.weight, diffusion.denoising.out_blocks.13.0.shortcut.bias, diffusion.denoising.out_blocks.14.0.conv_1.0.weight, diffusion.denoising.out_blocks.14.0.conv_1.0.bias, diffusion.denoising.out_blocks.14.0.conv_1.2.weight, diffusion.denoising.out_blocks.14.0.conv_1.2.bias, diffusion.denoising.out_blocks.14.0.norm_with_embedding.norm.weight, diffusion.denoising.out_blocks.14.0.norm_with_embedding.norm.bias, diffusion.denoising.out_blocks.14.0.norm_with_embedding.embedding_layer.1.weight, diffusion.denoising.out_blocks.14.0.norm_with_embedding.embedding_layer.1.bias, diffusion.denoising.out_blocks.14.0.conv_2.2.weight, diffusion.denoising.out_blocks.14.0.conv_2.2.bias, diffusion.denoising.out_blocks.14.0.shortcut.weight, diffusion.denoising.out_blocks.14.0.shortcut.bias, diffusion.denoising.out.conv.weight, diffusion.denoising.out.conv.bias, diffusion.denoising.out.gn.weight, diffusion.denoising.out.gn.bias, diffusion.ddpm_loss.norm_factor 2023-11-08 13:30:24,918 - mmgen - INFO - Try to load Tero's Inception Model from 'work_dirs/cache/inception-2015-12-05.pt'. 2023-11-08 13:30:25,052 - mmgen - INFO - Load Tero's Inception Model successfully. 2023-11-08 13:30:25,090 - mmgen - INFO - FID: Adopt Inception in StyleGAN style 2023-11-08 13:30:25,145 - mmgen - INFO - Load reference inception pkl from work_dirs/cache/chairs_test_inception_stylegan.pkl 2023-11-08 13:30:25,201 - mmgen - INFO - Sample 8 fake scenes for evaluation

However, considering that the "chairs" test dataset still yielded reasonable results, I believe this may not have an impact. I'm not sure if my understanding is correct.

If my data is not suitable, please let me know. After all, there is still a lot of noise compared to standard datasets. Looking forward to your response.

Lakonik commented 10 months ago

Hi! It seems that there could be an issue with your reconstruction setup, say the camera pose (we use OpenCV camera convention, and some data use Blender/OpenGL conventions). Usually, even if there's a failure in the actual 3D geometry, the result should still look close to the input at least from the given perspective.

  1. You can just use recons1v, these models are the same except for testing configuration.
  2. We have not rigorously tested the robustness against pose error, but from my experience it should be reasonably robust. I have a GUI demo under development, and for images in the wild I just align the camera pose manually in the GUI (may be released around the end of this year or), which is not accurate but the results are still good enough most of the time.
  3. If your camera has distortion, it would be better to rectify the image and obtain the correct intrinsics before feeding it to SSDNeRF. But again, sometimes you can set the intrinsics by manually tweaking the focal length and the model should be reasonably robust to this error.
  4. This is not an error, I just haven't suppressed these warnings yet.