ZHU-Zhiyu / NVS_Solver

Source code of paper "NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer"
245 stars 4 forks source link

How to test using my data set #9

Open d-irie opened 3 months ago

d-irie commented 3 months ago

Good Novel View images were obtained using the two image data (multi/scan1) provided in this repository. I think this is a great technology !!

Therefore, I would like to test using my image data.

To run svd_interpolate_two_img.py, I need to prepare camera parameters and monocular Depth as input data. Do you have any plans to provide scripts to prepare these data?

I also think that the code needs to be modified to process image data of arbitrary resolution. Do you have any plans to add tutorials or modify the code to process various data in the future?

mengyou2 commented 3 months ago

Thank you for your interest in our project.

The data structure is quite simple, you just need to put the required files, two images and their corresponding depth maps and camera parameters, into one folder. And make sure the depth map and camera parameter file share the same name with the image. In our experimental setting, we use the data from Tanks and Temples and DTU, thus the code provided shows how to read the information in these two dataset formats. If you would like to use your own data, you may need to change the code for reading camera parameters and depth or rewrite your data into the DTU format. As the sparse image setting requires the camera parameters, of which the format is quite different among different datasets, modifying the code to process various data can be a little complicated.

For the arbitrary resolution, you can just change the resized height and width in save_warped_image and add width and height params when calling the svd pipe. And you need to change the patch size in the optimization part, which is used for saving computing memory. You can refer to #6.

We'll release a more detailed instruction later. We appreciate your patience.