search

chaoyuaw / pytorch-vcii

Video Compression through Image Interpolation (ECCV'18) [PyTorch]
https://chaoyuaw.github.io/vcii/
GNU Lesser General Public License v2.1
209 stars 38 forks source link

Clarification on structure for optical flow files #7

Closed ajlangley closed 5 years ago

ajlangley commented 5 years ago

I have two questions about how to structure the optical flow files:

1) Each frame has 4 optical flow files associated with it: The x and y for the forward and backward flows, respectively. Does this mean the flow from frame n to n + 1 and the flow from frame n + 1 to n? Or does this mean we need the flow from frame n to n + 1, and n + 2 to n + 1?

2) If I'm using the calcOpticalFlowFarneback function from cv2 to produce the optical flow, is my understanding correct that this returns two arrays of pixels, one for the x and one for the y flows? Do you have any other recommendations for this?

chaoyuaw commented 5 years ago

  1. We need optical flow from "left" to "center" (to warp "left" so that it's similar to "center"), and optical flow from "right" to "center" (to warp "right" so that it becomes similar to "center". ) The (left, center right)-triplet is defined by our 12-frame hierarchical structure. That is, at the first level, (left, center, right) = (0, 6, 12). At the second level, its (0, 3, 6) and (6, 9, 12), and so on.

  2. Using calcOpticalFlowFarneback should work (I tried it before), even though it might have a different definition of "x" and "y" from what the current code assumed. So you might need some simple post-processing on that (e.g. things like swapping x and y, normalizing them into [-1, 1], etc. )

ajlangley commented 5 years ago

Thank you! That helps a lot.