How to get mask with optical flow

[tobymu]

@Deqing Sun
Thansk a lot. I have to get the mask with flow, but the mask result is not correct. Code in PWCNet.PY

def warp(self, x, flo):
        """
        warp an image/tensor (im2) back to im1, according to the optical flow
        x: [B, C, H, W] (im2)
        flo: [B, 2, H, W] flow
        """
        B, C, H, W = x.size()
        # mesh grid 
        xx = torch.arange(0, W).view(1,-1).repeat(H,1)
        yy = torch.arange(0, H).view(-1,1).repeat(1,W)
        xx = xx.view(1,1,H,W).repeat(B,1,1,1)
        yy = yy.view(1,1,H,W).repeat(B,1,1,1)
        grid = torch.cat((xx,yy),1).float()

        if x.is_cuda:
            grid = grid.cuda()
        vgrid = Variable(grid) + flo

        # scale grid to [-1,1] 
        vgrid[:,0,:,:] = 2.0*vgrid[:,0,:,:].clone() / max(W-1,1)-1.0
        vgrid[:,1,:,:] = 2.0*vgrid[:,1,:,:].clone() / max(H-1,1)-1.0

        vgrid = vgrid.permute(0,2,3,1)        
        output = nn.functional.grid_sample(x, vgrid)
        mask = torch.autograd.Variable(torch.ones(x.size())).cuda()
        mask = nn.functional.grid_sample(mask, vgrid)

        # if W==128:
            # np.save('mask.npy', mask.cpu().data.numpy())
            # np.save('warp.npy', output.cpu().data.numpy())

        mask[mask<0.9999] = 0
        mask[mask>0] = 1

        return output*mask

[jrenzhile]

@tobymu can you explain what you mean "get the mask with flow"?

[tobymu]

@luvegood Thanks for reply. I'm trying to reproduce some work in this paper https://arxiv.org/abs/1705.02092 As mentioned in Figure 5, a mask is estimated with optical flow between frame i and frame i+1. 0 in disoccluded regions and at the motion boundaries, and 1 everywhere else. I find that your code has done the warp operation, and I cannot understand what does mask defined in your code. Is it the same as this paper? Thanks!

[jrenzhile]

@tobymu it's two totally different things. The mask in Fei-Fei's paper means the occlusion map, and the "mask" used in this code is more of an implementational hack in PyTorch0.2 for the warping function.

[deqing]

@deqings , @tobymu would like to thank you 😄

[tobymu]

@luvegood I see, thanks!

[xuzhongyou]

@tobymu it's two totally different things. The mask in Fei-Fei's paper means the occlusion map, and the "mask" used in this code is more of an implementational hack in PyTorch0.2 for the warping function.

Could you explain the mask clearly in detail? I am also not clear about the implementational hack in PyTorch0.2.