search

LiUzHiAn / cv-utils

Basic image/video processing utilities in Python.
12 stars 3 forks source link

Direction of warping #1

Open Nianzhen-GU opened 3 years ago

Nianzhen-GU commented 3 years ago

I have a question about the warping direction. If I have the optical flow from img1 to img2 and I want to warp img2 back to img1, should I use img2 as the input image? Do I need to use the optical flow from img2 to img1 as the input optical flow?

shiqi1994 commented 3 years ago

I have the same question. Here is what I am thinking:

Say we use PWC-Net to estimate optical flow, input is img1 and img2, then let the output flow as u_{1->2}, the subscript denotes the flow is from img1 to img2.

then we calculate warp(img1, u_{1->2}) then we get warped img2.

Since estimation is not 100% accurate, so there should be some artifact in warped img2. ( like what we see in the example)

So back to your question, I think if one would like to warp img2 back to img1, then we need use the optical flow from img2 to img1.

shiqi1994 commented 3 years ago

I have a question about the warping direction. If I have the optical flow from img1 to img2 and I want to warp img2 back to img1, should I use img2 as the input image? Do I need to use the optical flow from img2 to img1 as the input optical flow?

Hi, have you solved the direction problem?

Nianzhen-GU commented 3 years ago

I have a question about the warping direction. If I have the optical flow from img1 to img2 and I want to warp img2 back to img1, should I use img2 as the input image? Do I need to use the optical flow from img2 to img1 as the input optical flow?

Hi, have you solved the direction problem?

I think it should be warp(img2, u_{1->2}). Because I think the algorithm is designed to warp the img2 back to img1 according to the optical flow from img1 to img2.

shiqi1994 commented 3 years ago

I have a question about the warping direction. If I have the optical flow from img1 to img2 and I want to warp img2 back to img1, should I use img2 as the input image? Do I need to use the optical flow from img2 to img1 as the input optical flow?

Hi, have you solved the direction problem?

I think it should be warp(img2, u_{1->2}). Because I think the algorithm is designed to warp the img2 back to img1 according to the optical flow from img1 to img2.

I think if one would like to warp img2 to img1, it is supposed to be warp(img2, -u(1->2)).

For example, say now we have img1, we do not know how pixels move since we only have this single image. Then we have img2( next frame of img1), if we can assign each pixel of img1 and img2, then we know how img1's pixels move so that img1 becoming im2.

If the optical flow is ideal, then img1 + flow = img2 (I am sorry it is not a rigorous notation, it only helps on understanding). Here we can see the flow is from img1-->img2.

Recall the definition of optical flow, it is the pixels' tiny displacement along the x and y axis. So if we set u{1->2} as the flow from img1 to img2, then u{2->1} = -u_{1->2}. ( such like \delta x is displacement along x axis while negative \delta x is displacement along the oppsite direction)

As a conclusion, given img1 as the first frame and img2 as the second frame, then we can estimate the optical flow u{1->2} (flow from img1 to img2); if one would like to warp img2 to img1, then calculate warp(img2, -u{1->2}).