search

manuelruder / artistic-videos

Torch implementation for the paper "Artistic style transfer for videos"
Other
1.75k stars 252 forks source link

Speed up processing using half resolution optical flow? #34

Open NameRX opened 7 years ago

NameRX commented 7 years ago

Hello!

Is there a way to use an optical flow generated in half-resolution to perform a style transfer for full resolution? Or maybe there is a way to upscale optical flow passes afterwards to full-resolution?

For example, optical flow calculations for 1 frame for 1280x720 takes ~3 minutes on my PC, but for 640x360 - ~1 min.

The reason i thought about this optimisation is that in video post-production software motion vectors can be calculated in half-resolution, and usually there is no a big seeing difference between using full-res optical flow and half-res.

Thank you!

NameRX commented 7 years ago

I discovered, that deepmatching-static has already using downscaling by 2 by default,

./deepmatching-static --help

Matching parameters: -downscale/-R <n=1> downsize the input images by a factor 2^n

So the solution is to change row 10 in run-deepflow.sh

./deepmatching-static $1 $2 -nt 0 | ./deepflow2-static $1 $2 $3 -match to ./deepmatching-static $1 $2 -nt 0 -downscale 2 | ./deepflow2-static $1 $2 $3 -match

That will force deepmatching to work with quarter resolution, and make the process really faster! So I need to run several tests to discover, is there a big difference between half and quarter optical flow resolutions, or not.

bododge commented 7 years ago

@NameRX I'd love to hear what you found out about using quarter res optical flow? Did it work well? Can you describe the process any further? I don't have a great handle on using the script yet, and would love any example you are willing to provide.

NameRX commented 7 years ago

@bododge The main differences, that I found, is that in half resolution intersections look better, object moving edges look more detailed and fast motions detecting better. But in practice, if you see only stylized result in quarter optical flow resolution, you will find this result satisfactory.

The way I solved this problem is to run optical flow calculations in background simultaneously with style transferring. Now there is no need to wait for first prepass, all processes run together, motion vectors on CPU and style transfer on GPU.

link to pull request - #37