psychosomaticdragon
commented
7 years ago One possibility i can think of off the top of my head for fast processing of a video file - use mpegflow (https://github.com/vadimkantorov/mpegflow) to extract the flow fields from a video file, scale them to the same size as the image frames you're processing - this takes the lengthy process out of the flow calculations, however the results will be much less accurate. Then, use a pretrained neural style network (https://github.com/jcjohnson/fast-neural-style) to stylize the image, while using the mpeg flow data to warp the image.
One of the problems with this approach is that you'll have to deal with i-frames in the video - ones in which there is no flow information, only image information. The two approaches i can see to this would either be interpolating the prior and post p-frame data to get approximate motion data, but this has it's own problems. Another approach would be to create a new stylized image at each i-frame, however this would likely create a stuttering effect in terms of the stylistic details present.
I can't think of any fast way to approximate the artistic-videos project that doesn't come at a severe cost to the video quality.
Is there a way to approximate the effect so it can be done in real time? For example you are familiar with the pixelization effect used to obscure the nasty private bits in video streams? Instead of a color averaged large block, one might use a ‘neural style’ thumbnail whose color average matches the calculated average of a 8×8 pixel sample in the image. The neural style thumbnail to which I refer is something that can be precalculated ahead of time from Deep Dream (https://en.wikipedia.org/wiki/DeepDream) So instead of getting a solid blob of color you get a neural pattern, in a video.