Film credit

[talosh]

Hi Hzwer, thank you for this great work!

I’d like to make you aware that RIFE-2020 is used for open-source timewarp toolset for Autodesk Flame: https://vimeo.com/505942142.

Despite being in early dev stage it has been successfully used in production of “Life in a Day 2020” documentary: https://www.youtube.com/user/lifeinaday that will premiere online at the Sundance Film Festival on February 1, and then stream on YouTube from February 6th. Please feel free to add it to the news.

There are several things I’d like to potentially discuss with you, one is an extension of an issue #91 to allow warping arbitrary number of channels while being driven by the first 3. This would come handy in many pro use cases. Would you be potentially interested in.

Another one is the amount of softness the image gets in some areas, is it something that is added to mask artefacts or does it come form training with perceptual loss? It would be handy to have some control on that aspect - in number of cases those errors are being addressed manually in pro vfx scenarios.

There are number of other things I’d like to discuss if possible, let me know if there’s interest for tuning it towards pro use and thank you again!

[hzwer]

Thank you for letting us know this information. We are still exploring and improving the effects of RIFE. I think any channel warping is not difficult to achieve, but some adjustments to the model and code are required. This function can be realized in one or two months. We are still studying how to reduce various artifacts by choosing an appropriate loss function.

If you have more suggestions, I would be happy to participate in the discussion. At present, I am mainly researching more suitable model structures and improving visual effects, which may last for several months.

[talosh]

Thank you for reply, it is really great to hear about potential improvements and the timing are fully understandable. There is two things I've noticed while working on that project and using RIFE with trained model v1.8 and I'd like to ask if it is something that might be potentially improved during train process in the future.

The first one is that it seem to create much better result on side edges of the frame then on top/bottom edges with camera panning. I did a quick test by getting the very same shot that has a vertical pan and rotating it 90 degree before interpolation and then comparing the result rotated back with non-rotated pass. You can check and see the difference on a video here: https://andriy.co.uk/examples/EDGES_Subclip_001.mp4 It looks that it might be potentially improved by adding random 90-degree rotation agumentation to the training process along with flipping and others, do you think it is correct?

Another one is that the model seem to catch larger motions much better with UHD flag passed (as I understood it brings flow res to a quater instead of default half) but obviously loosing the precision on finer details. Though most of the shows currently in production are between 3.2K and 6K in terms of horisontal resolution this effect is visible even on HD footage. I've recorded another video to illustrate this: https://andriy.co.uk/examples/UHD_Motion_Subclip_001.mp4 The question is it something that can be potentially improved by adding more larger multipliers during training process (say x8, x12, x16, x32) and maybe an otional two-pass workflow when model detects coarse motion in the first pass and then works on fine details on top of that pass?

[talosh]

Hi Hzwer, thank you v2.0 and v2.1 releases! It mostly seem to fix issues with edges and larger motion. Just one thing noticed doing side-by-side comparision is that v2.0 (based on RIFE_HDv2) seem to be slighty less robust againt repetetive patterns: https://andriy.co.uk/examples/fluid_18_vs20_vs21.mp4 v2.1 seem to be even slightly more loose with this, but overall - amazing improvement! Thank you again, Flame's pro community is really excited about this tool and it already helped many artists all over the world.

[hzwer]

Hi talosh, would you mind uploading this clip for me? I'll try to fix it next version.

[talosh]

Hi Hzwer, here is the link: https://we.tl/t-iRPFgX5IBK

The first is fluidmorph so I'm doing inference between incoming and outgoing shots without a ground truth. But this sort of camera movement could be emulated by getting a smaller window and moving it in x and y to simulate linear motion, say

dx = int(ranodm(-range, +range)) img0.shift_x = 0,
img1.shift_x = dx # ground truth img2.shift_x = dx*2

and same for y.

I'm not sure if you're already doing this type of augmentation but it seem to be good to add to simulate camera movements. There is another shot with moving train, when it gets to white carriages it is a good example of holding repetitive patterns as well. Thank you!