search

NVlabs / PWC-Net

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, CVPR 2018 (Oral)
Other
1.64k stars 355 forks source link

Question about the first optical flow estimator #63

Open Ecalpal opened 5 years ago

Ecalpal commented 5 years ago

Dear Dr. @deqings : I have read your paper(Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation) and code(pytorch). I found there are two inputs in the first optical flow estimator in the Fig. 2 in the paper. But in the code(https://github.com/NVlabs/PWC-Net/blob/master/PyTorch/models/PWCNet.py line422-431) corr6 = self.corr(c16, c26) corr6 = self.leakyRELU(corr6) x = torch.cat((corr6, self.conv6_0(corr6)),1) x = torch.cat((self.conv6_1(x), x),1) x = torch.cat((x, self.conv6_2(x)),1) x = torch.cat((x, self.conv6_3(x)),1) x = torch.cat((x, self.conv6_4(x)),1) flow6 = self.predict_flow6(x) up_flow6 = self.deconv6(flow6) up_feat6 = self.upfeat6(x) only corr6 fed into the first optical flow estimator.

Which is right? Or is there a special reason for not fed c16 into the first optical flow estimator?

Looking forward to your answer, thank you.

deqings commented 5 years ago

Sorry for the very late reply.

It is a good observation and a great question. My thought was that the correlation/cost volume might be sufficient for this level, but I didn't do a comparison with feeding c16 to the flow estimator. It would be an interesting experiment. I expect there may be some performance gain as the model is bigger, although I am not sure how much the gain would be.

Ecalpal commented 5 years ago

@deqings Thanks for your reply! I have another question to ask. As everyone knows, PWC-Net has become the new baseline of many papers. I wonder if its pytorch version of the training code is still in progress？ I tried to reproduce the results of my paper on pytorch from scratch， but it seems that data enhancement is the bottleneck. Do you have any suggestions for novices learning optical flow?

Thanks again.