search

sniklaus / softmax-splatting

an implementation of softmax splatting for differentiable forward warping using PyTorch
468 stars 58 forks source link

Why use forward warpping? #64

Open JiahaoChen1 opened 23 hours ago

JiahaoChen1 commented 23 hours ago

Thanks for your work! The softmax-splatting is used for forward warping. Since both forward and backward warping need to estimate the optical flow for VFI, what is the advantage of using forward warping over backward warping?

sniklaus commented 20 hours ago

Thank you for your interest in our work! Given the optical flow from a frame A to a frame B, one can use backward warping to warp frame B to frame A but one cannot just warp B halfway to A which would be necessary for interpolation. In contrast, one could easily warp frame A to any position in between the two frames using forward warping. If one wanted to use backward warping in this scenario, it would be necessary to first estimate the flow from the not-yet-existent in-between frame to frame A/B. Some papers do this, from the top of my head Voxel Flow was the first one, but it comes with a bunch of limitations.