search

xinntao / EDVR

Winning Solution in NTIRE19 Challenges on Video Restoration and Enhancement (CVPR19 Workshops) - Video Restoration with Enhanced Deformable Convolutional Networks. EDVR has been merged into BasicSR and this repo is a mirror of BasicSR.
https://github.com/xinntao/BasicSR
1.49k stars 318 forks source link

Another problem in VSR #35

Closed splinter21 closed 5 years ago

splinter21 commented 5 years ago

How about subtitle region ？ There is no motion in subtitle region, but it changes instantaneous between two frames. Example figure(Average between 5 frames. The head region can be aligned, but the subtitle region can't be aligned.): avg

Now we can't label the subtitle region, but the model should be robust for these kind of situations.(Another situation is for anime videos(2D painting). Just use SISR methods for SR.) Will it influence the training when there is subtitle region included in the input patches of network?

This problem is caused by the diffenrence between experimental environment in academic research and practical application. In ideal experimental datasets, subtitle is not included.

splinter21 commented 5 years ago

Maybe I can calculate the optical flow between center frames to get six maps, and use these six maps to kill the subtitle region and static region in neibor frames（these are the preprocess）. What's your opinion? Will this be helpful to learn more motion details from neighbor frames?

xinntao commented 5 years ago

I think using SISR for subtitle is more appropriate because there is little information from other frames in the subtitle region. Is it possible to use some simple tricks to detect the subtitle regions and then process these regions separately?