How to run the project in custom dataset?

[VikasXitadel]

Hi, thanks for your great work. I've trained my model on the TNT dataset, focusing on the playground data. After training, I've tested it with smog and snow. Now, I'm eager to apply the same procedure to my custom dataset. Can you help me understand the training process and how to generate the checkpoint?

[y-u-a-n-l-i]

I‘ve added a dataloader for Colmap processed dataset in MipNeRF360's format. Here is a breakdown of the directory structure:

├── images
├── images_2
├── images_4
├── images_8
├── poses_bounds.npy
└── sparse

poses_bounds.npy is not necessary in our code. You need to run Colmap for your captures to get cameras.bin, images.bin and points3D.bin in sparse/0. Be careful when you are trying to load down-sampled images for training. For example, you need to store 4-times downsampled images in images_4 folder. After that, you need to parse colmap to --dataset_name to enable the dataloader for Colmap during training and all the remaining steps. You can refer to this line in opt.py for all the supported dataset types. You can also refer to the config file of Garden scene in MipNeRF360 dataset for how our experiments are conducted on Colmap processed datasets.

[VikasXitadel]

Hi, thanks for the reply. As you suggested, I tried running COLMAP on my custom video to generate the cameras.bin, images.bin, and points3D.bin files in the sparse/0 directory. I managed to generate four folders named images, images_2, images_4, and images_8, along with a colmap directory containing a sparse directory which is empty. However, I am unable to generate the three bin files you mentioned. The process is showing an error that no good initial image pair is found.

My video features pure forward motion, and I discovered that pure forward motion is problematic for structure-from-motion as it leads to unstable (and degenerate) configurations. I am stuck at this point. If you have any idea about how to generate the bin files, it would be a great help to me.

Below is the link to my dataset.

https://xitadel2-my.sharepoint.com/:u:/g/personal/amrithavikas_krishnakumar_xitadel_com/EfO89fyqnbRJqlkz4vYgPcMBCPBngRFEtd82ksaOKeBVJA?e=yM3hTI

Thanks in advance.

[VikasXitadel]

Hi @y-u-a-n-l-i , I hope you are doing well. Following your suggestion to work on a custom dataset using Climate NeRF, I took images from the Kitti-360 data provided in the readme section and generated the dataset in the specified format using Colmap. After adjusting some reconstruction options, I successfully generated the bin files and sparse directory with Colmap. However, when I used this dataset to train the model with your training code, the reconstruction quality was not as expected compared to the original Kitti-360 dataset result. I will attach the render_traj.mp4 video output that I obtained when I trained the model with the dataset I built using Colmap.

Below is the link to my video.

https://xitadel2-my.sharepoint.com/:v:/g/personal/amrithavikas_krishnakumar_xitadel_com/EbRpvNlPeYhLpBjna3tO_mQBXes65OUgYEBwx_u_qnIxAA?nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJPbmVEcml2ZUZvckJ1c2luZXNzIiwicmVmZXJyYWxBcHBQbGF0Zm9ybSI6IldlYiIsInJlZmVycmFsTW9kZSI6InZpZXciLCJyZWZlcnJhbFZpZXciOiJNeUZpbGVzTGlua0NvcHkifX0&e=8sLjje

Any assistance would be greatly appreciated.

[y-u-a-n-l-i]

Sorry for the late response. It seems that I don't have the right to access your link. It is recommended to use the dataloader designed for KITTI-360 if you want to run experiments on that dataset since we load KITTI-360's ground truth depths as geometry supervisions. You can find the example config file here and organize KITTI-360 data according to this. python train.py --config configs/kitti360_1538.txt is the command that I used to train our model.

[VikasXitadel]

Hi @y-u-a-n-l-i ,

Thanks for your response. I'd like to clarify a few points:

1. I'm not working with the KITTI dataset. I have already trained a model using the KITTI-360 dataset following this approach.

2. What I did with the KITTI dataset was to use its images to create an images directory and bin files with COLMAP. However, whenever I tried to train the Climate NeRF model with the dataset I prepared using COLMAP, it didn't work. I will attach the video below.

https://github.com/y-u-a-n-l-i/Climate_NeRF/assets/169804565/f0392887-4b1f-4d9d-b2fa-3f4581eb2973

3.My main concern is running Climate NeRF on my custom dataset, which is very similar to the KITTI-360 dataset. I generated the images directory and bin files using Colmap as per your last reply, but whenever I try to train the model, I can't get a well-reconstructed video.

Any assistance would be greatly appreciated.

[y-u-a-n-l-i]

It should be related to the scene scale parameter. Or Colmap failed to reconstruct camera poses on your sequence. Can you share your config file and your running command? Sending your processed data to me also helps. Also, if you are using Colmap gui, you might visualize and check the camera poses and sfm point cloud before running nerf code.

[VikasXitadel]

Hi @y-u-a-n-l-i, I am attaching a drive link where you can find the datasets (4 image directories and the sparse directory created using COLMAP), the configuration file, and the commands I used to train the model.

https://drive.google.com/drive/folders/17l5JT5KncQewyK-zukz-Dd2GJlpCcPoz?usp=sharing

Thanks in advance for your help.

[y-u-a-n-l-i]

It seems that you mix all the sequences from different scenes. You have to separate the sequences and reconstruct them independently using Colmap and NeRF. nerf_pl is a good starting point to learn NeRF.

[VikasXitadel]

Hi @y-u-a-n-l-i, Thanks for your reference. I took the dataset from this source. Here is what I did to generate the images and bin files:

1. I specifically used images from the image_00 directory and provided them as input to the NerfStudio command to generate the images directory (images, images_2, images_4, images_8). I used the following command from NerfStudio for that:

ns-process-data images --data C:/Data_Science/Vikas_Projects/Custom_kitti/images/ --output-dir C:/Data_Science/Vikas_Projects/Custom_kitti/output/

2. Since the above Nerfstudio command didn't generate the bin files (it showed "no image pair found"), I edited the reconstruction options available in the COLMAP GUI, particularly for forward motion, which in turn generated the bin files inside the sparse directory. Here is a screenshot of the settings:

3. From the above process, I got the image directories from the first step and bin files required from the second step and used them as the dataset for training.

Thanks in advance for your help.

[y-u-a-n-l-i]

But have you ever really checked the images inside the dataset? You are trying to reconstruct images from different scenes in just one Colmap and NeRF model. This might lead to the failure of your optimization. There is no problem with your Colmap and image operations.

[y-u-a-n-l-i]

You can refer to this processed data by Colmap. In the images folder, there are only images from 1538 to 1601. Directly running our code on this data gives you rational results, but reconstructions will be better if you have lidar depth or use stereo images like the KITTI-360 dataset.