search

facebookresearch / localrf

An algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video.
MIT License
976 stars 60 forks source link

About transforms.json #10

Closed Khoa-NT closed 1 year ago

Khoa-NT commented 1 year ago

Hi, thank you for sharing the code of this amazing research. Can I ask what is the difference (or use case) of each 3 transforms.json files after finishing the training?

  1. output/forest1/smooth_spline/transforms.json
  2. output/forest1/transforms_rf.json
  3. output/forest1/transforms.json

It confused me because you also have another one in the dataset: hike_scenes_localrf/forest1/transforms.json (4)

I guess (4) is created from colmap and (1) is used for rendering the video.

ameuleman commented 1 year ago

Hi, You are correct. Also, (2) is each radiance field's pose and (3) contains the estimated camera poses.

Khoa-NT commented 1 year ago

Hi, thank you for the clarification. Then I can use (3) as camera poses for training another NeRF model.

I'm curious about (2), is it the local camera poses for the Local radiance fields in sec 3.3 in the paper?

ameuleman commented 1 year ago

Hi, Yes, specifically the t_j in eq. (7).

rockywind commented 1 year ago

@ameuleman @Khoa-NT How to generate the transform.json on our own dataset? Is there any tutorial?

ameuleman commented 1 year ago

@ameuleman @Khoa-NT How to generate the transform.json on our own dataset? Is there any tutorial?

Hi, transforms.json files in our dataset correspond to COLMAP poses. We run COLMAP using MultiNeRF's script. Note that transforms.json is not needed to optimize LocalRF, except with the argument --with_preprocessed_poses 1.

rockywind commented 1 year ago

@ameuleman @Khoa-NT How to generate the transform.json on our own dataset? Is there any tutorial?

Hi, transforms.json files in our dataset correspond to COLMAP poses. We run COLMAP using MultiNeRF's script. Note that transforms.json is not needed to optimize LocalRF, except with the argument --with_preprocessed_poses 1.

Hi, Thanks for quick replay!

  1. I want to run the localRF code on autonomous driving datasets such as KITTI and nuScenes, which include pose information with some noise. LocalRF can optimize the initial noisy pose from the autonomous driving datasets.
  2. Are the poses from COLMAP in the OpenCV coordinate system? I noticed the poses in NerfStudio are based on OpenGL coordinates.The below picture show the OPENCV and OPENGL coordinate. image
ameuleman commented 1 year ago

Oh right, there is a conversion required after COLMAP. Instant-NGP's script can achieve it from COLMAP outputs.

rockywind commented 1 year ago

Thanks a lot!