search

lmb-freiburg / flownet2

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
https://lmb.informatik.uni-freiburg.de/Publications/2017/IMKDB17/
Other
1k stars 318 forks source link

Shadow suppression #217

Closed DonaldKam closed 4 years ago

DonaldKam commented 4 years ago

It a marvelous job and I noticed that the motion of shadow was suppressed in the supplementary video shown as following image This differentiates from conventional methods like Lucas-Kanade. Can you explain why？

nikolausmayer commented 4 years ago

There is no explicit shadow treatment in FlowNet2 — every behavior is data-driven. The network has enough capacity to learn something about shadows from the available KITTI groundtruth during finetuning (after pretraining on a larger dataset to learn general flow features). FlowNet2 has a relatively large receptive field, so it can "analyze" the area around the car and shadow and make decisions based on that. Classical methods, especially purely local ones like Lucas-Kanade, do not learn correlations between appearance and flow. Consequently, they can never distinguish a black object from a black shadow. A neural network can learn that if black blobs and objects often appear together, then a black blob could be a shadow.