search

visinf / irr

Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation (CVPR 2019)
Apache License 2.0
194 stars 32 forks source link

Function rescale_flow modifies input variables #17

Closed tomas2211 closed 4 years ago

tomas2211 commented 4 years ago

Hi, the function rescale_flow that converts the scale of optical flow between global and local scale changes also the input flow variable (besides returning the rescaled flow). Operations done on the input flow of the function - chunk and multiplication - are both in-place.

https://github.com/visinf/irr/blob/3c0468b1b9d24d43d3f758be709b912f58b94229/models/pwc_modules.py#L47-L59

This wouldn't be a problem because the function is usually used with the same input and output variable as flow=rescale_flow(flow,...).

However, it is not the case in IRR_PWC model on lines 125 and 126. Flow in variables flow_cont_f, flow_cont_b gets rescaled here to the "global" scale for the warping module. Nevertheless, the flow gets rescaled once more on lines 132 and 133.

https://github.com/visinf/irr/blob/3c0468b1b9d24d43d3f758be709b912f58b94229/models/IRR_PWC.py#L125-L133

The loss is evaluated on this double-rescaled flow. When using a pre-trained model, the flow flow_cont_f is thus correctly in the global scale after rescaling on line 132 and it is in the local scale after rescaling on line 125 until 131. It thus enters the warping function in the local scale (but the global scale is expected), which causes the img2_warp to be an incorrect warp of img2_resize. As a consequence, the bilateral refining module gets an incorrect input img1_resize - img2_warp.

Furthermore, flow_cont_f is in an incorrect scale above the line 125 which makes decoder and context modules compensate for this.

Have a nice day, Tomas

hurjunhwa commented 4 years ago

Dear Tomas,

I truly appreciate that you found this out and shared it here. What you are saying is indeed right!! It was a mistake indeed :(. I will re-run the IRR-PWC experiments without using the in-place operation in rescale_flow(), and let you know the results.

Best, Jun