search

ZHU-Zhiyu / NVS_Solver

Source code of paper "NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer"
242 stars 4 forks source link

What is the unit of displacement for the new perspective? #21

Open Antonio521 opened 1 month ago

Antonio521 commented 1 month ago

I want to generate a new view by moving one meter to the left

mengyou2 commented 1 month ago

We don't have an absolute scale, as we use depth maps from monocular depth estimation. If you wanna try the circular trajectory and make the center object, such as the sculpture in the example image, always in the center of the generated video, the minor radius should be the depth value of the center object. The major radius determines how far the trajectory goes to right/left and you can calculate the value corresponding to 1m according to the ratio of absolute depth/ relative depth if you know the absolute depth value of the center object.

Antonio521 commented 1 month ago

Your paper demonstrates the use of depthanything for depth estimation, with units in meters. However, I have inverted and limited the depth in your warp, making it difficult to align the units for subsequent pose generation. I don't want to focus around the center of my field of view, I just want to obtain the translated pose. I also tried aligning with the true depth ratio, but the obvious effect was not good.

 

------------------ 原始邮件 ------------------ 发件人: "ZHU-Zhiyu/NVS_Solver" @.>; 发送时间: 2024年7月22日(星期一) 下午4:47 @.>; 抄送: "XinLi @.**@.>; 主题: Re: [ZHU-Zhiyu/NVS_Solver] What is the unit of displacement for the new perspective? (Issue #21)

We don't have an absolute scale, as we use depth maps from monocular depth estimation. If you wanna try the circular trajectory and make the center object, such as the sculpture in the example image, always in the center of the generated video, the minor radius should be the depth value of the center object. The major radius determines how far the trajectory goes to right/left and you can calculate the value corresponding to 1m according to the ratio of absolute depth/ relative depth if you know the absolute depth value of the center object.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

mengyou2 commented 1 month ago

In the single view setting, the depth value and also the camera parameters are not absolute values. So if you limiting depth to the real value, you also need to change the camera parameters in the warp code to real value.

Antonio521 commented 1 month ago

Thank you for your explanation. When I was processing it, I found that after reading depth images, depth = 10000./depth depth = np.clip(depth,0.0001,500)

Can you explain the purpose of the above processing? Especially the value 10000, Why did you use 10000?

 

------------------ 原始邮件 ------------------ 发件人: "ZHU-Zhiyu/NVS_Solver" @.>; 发送时间: 2024年7月22日(星期一) 晚上10:27 @.>; 抄送: "XinLi @.**@.>; 主题: Re: [ZHU-Zhiyu/NVS_Solver] What is the unit of displacement for the new perspective? (Issue #21)

In the single view setting, the depth value and also the camera parameters are not absolute values. So if you limiting depth to the real value, you also need to change the camera parameters in the warp code to real value.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

mengyou2 commented 1 month ago

The DepthAnything outputs disparity, so we need first calculate the inverse value. The 10000 is picked depending on the range of the disparity to get the depth in a normal range like 1-1000.

btw, if you want make the camera center parallel to the trajectory, you need to change the z value in https://github.com/ZHU-Zhiyu/NVS_Solver/blob/af820d19bb1b71a3d40a8cd0bcdf2cd2d6f05ce3/svd_interpolate_single_img_traj.py#L1021 to the corresponding position of how far moved to left from the origin.