Motion blur magic

[AdrianEddy]

We could implement a way to remove the motion blur using AI model: https://github.com/google-research/maxim

Then using the new AI-based optical flow, we can add the motion blur back after stabilization

For reference:

• https://github.com/bevyengine/bevy/pull/9924/files
• https://github.com/phiresky/opencl-motion-interpolation
• https://github.com/OpenCL/GEGL-OpenCL/blob/master/opencl/motion-blur-linear.cl

[webben216]

If I understand correctly, do you want to process all frames with the neural network or only certain frames with a certain amount of blur? The example on replicate takes around 17 seconds for a single frame on a Nvidia T4 GPU, im my opinion way to long to process all frames.

Looking at my footage with long exposure times (walking), only around 10% of the frames have significant motion blur.

Things I am asking myself:

• How bad is the flickering on successive frames? Do we need some temporal filtering?
• How does the model deal with 10, 12, or 14 bit raw/ log footage?

[AdrianEddy]

Only some frames, the ones with most blur, based on the movement amount from gyro data. Each frame will likely need to be stored to a file. Hard to tell exactly how it will behave in real world, but I imagine we can add some blending or something if there's too much inconsistency.

I'm sure the network will be much faster in Rust than in Python. Python does a lot of slow things before and after actual neural network, we'll do all of them on a GPU in a single step instead.

[webben216]

That makes sense. I never used cog, but with TensorRT, the inference time didn't change significantly between the Python and C++.

Why do we need to add motion blur back in Gyroflow? In my view, motion blur is a totally separate feature. The only advantage I see adding motion blur in Gyroflow is a reduced crop, if it is possible to apply it to the full unstabilized frame with the information of the stabilization.

[AdrianEddy]

Because motion blur looks good if it's done right. That's why people use ND filters and slow shutter speeds, but this is incompatible with digital stabilization, so we can add it artificially after stabilization.