Question on OpticalFlow to Node motion

[changhaonan]

Congrats on your great work! I am currently trying to re-implement your work. I am faced with a problem: you said in your paper that you generate 3D notion motion from optical flow image. I currently can think up of two ways to do it:

1. Project node position to the optical flow image, and read value at that pixel.
2. Compute the motion of each vertex, and compute node motion by averaging the motion of its nearby vertex.

Which one should I choose?

Best, Haonan

[wenbin-lin]

Thanks for your interest.

We first project the position of the 3D node into the 2D image space and read the depth value of the projected pixel. If the depth value matches the depth of the 3D node, we further read the optical flow value of the pixel to calculate the motion of the node.

In addition, to reduce the effect of optical flow errors, we calculate the average 3D motion of a small area around the projected pixel as the motion of the node.

Our approach is closer to the first way you mentioned, but the second way is also reasonable.

[shubhMaheshwari]

Can you elaborate on how the average motion is calculated ? Do you use something like torch.nn.funcational.grid_sample or you take a 3x3 window and average the values in that window ?

[wenbin-lin]

We take a 7x7 window to calculate the average motion, and the projected pixel is the center of the window.

[shubhMaheshwari]

Is the above averaging also used to find visible vertices in vertex pair set (P) during optimisation ?

[wenbin-lin]

For finding the vertex pair set P, we do not use the above averaging method.

[NothingL]

I also got a question about how to calculate the motion of the node, since optical flow only provides 2d directions, there is missing on the z-axis, so how to correctly project it to the 3d node? @wenbin-lin