search

jasonyzhang / ners

Code for "NeRS: Neural Reflectance Surfaces for Sparse-View 3D Reconstruction in the Wild," in NeurIPS 2021
https://jasonyzhang.com/ners
BSD 3-Clause "New" or "Revised" License
299 stars 32 forks source link

Creating metadata.json file for my own dataset #7

Open AliKaramiFBK opened 2 years ago

AliKaramiFBK commented 2 years ago

Hi I want to run this net on my own dataset I have images and masks also the calibration file that I got from metashep including interior and exterior orientations but I dont know how to create metadata.json file as it is required to run your code

would you please let me know how can I create this file

best regards Ali

jasonyzhang commented 2 years ago

Hi Ali,

To use NeRS out of the box, you will need to set initial cuboid dimensions and the camera extrinsics via azimuth/elevation representation in degrees. The metadata.json is just an easier way to store this information. I would recommend that you start with the demo notebook which you can also run on colab. The notebook will show how these values are used.

To set the camera parameters using your calibration data, you will need to convert the extrinsics to rotation matrices. What is the format of your extrinsics data?

Also, how many images are you using? Typically, for sparse views, automatically recovered camera poses are highly inaccurate.

Best, Jason

AliKaramiFBK commented 2 years ago

Hi Jason thanks for your reply I have the calibration file in Metashape and also calmap so I can have many different formats but not metadata.json do you know how can I convert it ? any libraries or available scripts to do that. Here are some formats that I can get from Metashape. image

Best Ali

jasonyzhang commented 2 years ago

Hi,

You will need to convert your camera poses to the Pytorch3D camera convention.

Given the 4x4 camera extrinsics tensor from COLMAP, you should be able to convert it to the Pytorch3D camera convention by doing something like this (based on https://github.com/facebookresearch/pytorch3d/issues/1120):

T = np.array(colmap_camera)  # 4x4
torch3d_T_colmap = np.array([[-1, 0, 0], [0, -1, 0], [0, 0, 1]])
R, t = T[:3, :3], T[:3, 3]
R = (torch3d_T_colmap @ R).T
t = torch3d_T_colmap @ t

If you want to use the COLMAP cameras, you should no longer being using metadata.json, but rather write your own loader that reads these camera poses.

How many images are you using? What is the output when using the demo notebook with hand selected poses?

eric-yim commented 1 year ago

Can you link sample code for loading camera poses from COLMAP? I'm lost with how to start.