search

philferriere / tfoptflow

Optical Flow Prediction with TensorFlow. Implements "PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume," by Deqing Sun et al. (CVPR 2018)
MIT License
525 stars 133 forks source link

Unsupervised training #24

Open martinv opened 5 years ago

martinv commented 5 years ago

Hello Phil,

thank you for this very nice implementation of PWC-Net.

I have a question about unsupervised training using image pairs only. If I wanted to implement this, is there anything else besides the loss function I need to modify in the source code in order to make it work or do you think a proper loss formulation should be sufficient?

Thank you.

Kind regards,

Martin

fperezgamonal commented 5 years ago

Hello @martinv , I am currently doing my MsC thesis in Optical Flow interpolation and I've taking a look at testing unsupervised learning for OF estimation too. That is why I think you may find a very recent paper (SOTA on Sintel) quite interesting and helpful to do what you are asking. The method is called [Code][Article] and uses self-supervised learning to estimate Optical Flow very cleverly (avoiding completely the use of FlyingChairs or FlyingThings3D).

The training code will be released this month according to the paper's main author and their architecture is based on PWC-Net+ and is written in Tensorflow too. As for lost, they use the Photometric Loss for non-occluded pixels and a combination of Photometric + Self-supervised for a later refinement.

Unfortunately I did not have the time to look into the testing code as of yet but I am eagerly waiting to train it and play with it. Hope this is helpful!

Cheers,

Ferran.