search

owl-project / NVISII

Apache License 2.0
328 stars 28 forks source link

Is there any way to generate optical flow from motion vector? #138

Closed YWQQQQQQ closed 2 years ago

YWQQQQQQ commented 2 years ago

Example "20.motion_vectors.py" renders and visualizes a 4-dim motion vector but what's the meaning of each dimension in motion vector? I couldn't find it in documents and tutorial. Is there any way to generate 2-dim optical flow (horizontal flow & vertical flow) from motion vector? Thanks!

natevm commented 2 years ago

Our motion vectors are the same type of motion vectors used by TAA or DLSS. They are returned as vec4s because most image formats expect four channels, but you only need to read the first two channels with motion vectors. These first two values indicate the motion of the object in normalized screen space. In the motion blur example you brought up, we scale those motion vectors to account for resolution, and then use those motion vectors to lookup object locations in the previous frame, as well as to detect disocclusions.

The main difference between motion vectors and optical flow is that optical flow accounts for apparent motion, while motion vectors account for geometric motion independent of material. Otherwise, they both give horizontal and vertical flow and can often be substituted for each other.