Mode switching and warping.

ZHU-Zhiyu / NVS_Solver

Source code of paper "NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer"

245 stars 4 forks source link

Mode switching and warping. #3

Closed bopan3 closed 3 months ago

bopan3 commented 3 months ago

Hi, Nice work! I wonder how to swtich between the DGS mode and Post mode? Moreover, I wonder how to let warping code work for zooming out? When I try to zoom out the camera with current warping code, there are obvious aritfacts caused by inccorect zoom out.

ZHU-Zhiyu commented 3 months ago

Hi, @bopan3 , the current version only contains NVS-Solver (Post). We will release the NVS-Solver (DGS) in the next few days. For the arbitrary camera trajectory as zoomed out, we are also preparing the code. (Actually, due to the large downsampling ratio of VAE, a straightforward solution is to apply more erosion to the warped image.) We appreciate your patience.

bopan3 commented 3 months ago

I see, thanks!

mengyou2 commented 3 months ago

Hi, I've identified the cause of the artifacts during zoom out. The issue was due to a depth normalization step using $depth = depth - depth.min()$ before warping. Removing this step eliminates the artifacts. I've uploaded the new version, and you can try zooming out now.

bopan3 commented 3 months ago

That's great!