search

Doubiiu / DynamiCrafter

[ECCV 2024] DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
Apache License 2.0
2.09k stars 165 forks source link

Frames sampling with different FPS #34

Closed szxiangjn closed 3 months ago

szxiangjn commented 4 months ago

Hi,

Thanks for your wonderful work! I don't quite understand the video frames sampling process. You mentioned in the paper that FPS is sampled between 5~30, so my understanding is that you first sample an FPS and then sample frames from a video by the FPS. Say like, we have a 10s video and an FPS of 16, how do you sample frames from the video?

Best

Doubiiu commented 4 months ago

Hi. We did that in a simple way: given a video clip in dataloader, we fetch its fps meta information and then sample frames with a stride frame_stride (e.g., 1-6), and then calculate the actual fps after sampling and pass this fps to the model.

szxiangjn commented 4 months ago

Since the frame_stride and frame number (16) are fixed, you may only sample from part of the video (e.g., with frame_stride as 6, the latest frame we can get is the 96th frame, thus for a video with 192 frames you are only sampling from the first half of the video), is that correct?

Doubiiu commented 4 months ago

Exactly.

szxiangjn commented 3 months ago

Thanks!