search

DQiaole / MemFlow

[CVPR 2024] MemFlow: Optical Flow Estimation and Prediction with Memory
Apache License 2.0
114 stars 6 forks source link

Can we run next frame inference with your demo? #3

Open tykim5931 opened 4 months ago

tykim5931 commented 4 months ago

Hello, thank you so much for your fascinating work. :) I'm currently working on a project which requires next frame inference, and I want to use your pretrained model for frame generation. As current inference.py code predicts optical flow only, I wonder if it is possible to run next frame prediction with your demo. Thank you.

DQiaole commented 4 months ago

Hi, thanks for your interest in our work. You can combine our predicted optical flow with softsplat and our image inpainting model for next frmae synthesis. Combining these codes and models into a single repository can be overwhelming. Therefore, we do not directly provide a demo.

reallybigname commented 4 months ago

I'm gonna try to get this thing running! I'm one of the developers on Deforum, and I've been trying to find some type of flow prediction. We can use a predicted flow, remap the image with it, then send it back through stable diffusion. Although, I bet your tricky way could be better! I may check those other tools out too though - softsplat seems interesting all by itself! I may be able to work something out with inpainting in Deforum as well. I'm gonna look for more examples of this stuff working!