search

MyNiuuu / MOFA-Video

Official Pytorch implementation for MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model.
https://myniuuu.github.io/MOFA_Video
Other
358 stars 22 forks source link

Questions with hand-crafted trajectories #5

Closed songwoh closed 3 weeks ago

songwoh commented 3 weeks ago

First of all, thank you very much for sharing such a wonderful work!

I have a question with the formulation of hand-crafted trajectory. If I understand correctly, hand-crafted trajectories F_{i-1} are sparse motion hints between frames i and 0. However, according to the definition of F (from section 3.2, ), F^s_i is a sparse forward optical flow between frames i and i+1. If the network was trained on the sparse optical flow samples from the definition from 3.2, wouldn't this mismatch between the two definitions cause unexpected behavior, since definition for F for hand-crafted trajectory is vastly different from what the network was trained on?

Let me know if I am missing something. Thanks!

MyNiuuu commented 3 weeks ago

Hi, thank you for your interest in our project.

In section 3.2, within the 'Sparse Motion Vectors from Dense Optical Flow' part, there is a typo in the definition of F. The text states that "F_i represents the flow sequences from the i-th to the (i + 1)-th frames." However, F_i should actually represent the optical flow from the 0-th to the i-th frames.

We appreciate your notice of this issue.