Garbage results with frame count <16

[phr00t]

Is it possible to generate interpolation at <16 frames? I see "up to 16 frames", but any time I try anything lower than 16 frames of interpolation, I get garbage:

This speeds up generation far more than lowering step count and I'd love to be able to make this work!

Related on the wrapper github: https://github.com/kijai/ComfyUI-DynamiCrafterWrapper/issues/9

[Doubiiu]

Hi. Thanks for your interest. Since the model was trained with the fixed temporal frame length (16), the performance of directly inferring frames<16 may be degraded. You can try to infer 16 frames but select the required frames with strides. BTW, as an extension application of the single image animation/image-to-video proposed in the paper, currently the extension interpolation model was not trained on the setting that two input frames are overly different (as shown in your workflow), the success rate may be lower than your expectation. Lastly, I find the model is doing well when the camera motion is involved (zooming, panning, rotating, etc.) Thanks for your interest again. Looking forward to your great creation of videos.

[phr00t]

Hi. Thanks for your interest. Since the model was trained with the fixed temporal frame length (16), the performance of directly inferring frames<16 may be degraded. You can try to infer 16 frames but select the required frames with strides. BTW, as an extension application of the single image animation/image-to-video proposed in the paper, currently the extension interpolation model was not trained on the setting that two input frames are overly different (as shown in your workflow), the success rate may be lower than your expectation. Lastly, I find the model is doing well when the camera motion is involved (zooming, panning, rotating, etc.) Thanks for your interest again. Looking forward to your great creation of videos.

Thank you for the response!

Here you can see I am using a "frame stride" of 10, which was the default value. I don't really know how that parameter works, but it sounds like I'd still need to infer 16 frames with different frame stride values, so I wouldn't save any time, correct?

Even though the frames are quite different, the animation was halfway decent when doing it @ 16 frames:

https://github.com/Doubiiu/DynamiCrafter/assets/5983470/fad36203-b6ad-4ba0-8fb1-9188588d565f

[Doubiiu]

Here you can see I am using a "frame stride" of 10, which was the default value. I don't really know how that parameter works, but it sounds like I'd still need to infer 16 frames with different frame stride values, so I wouldn't save any time, correct?

Larger fs normally means the generated motion magnitude could be smaller (more stable results), and vice versa. Yeah you need to infer 16 frames anyway (but more frames are possible I believe?) and cannot save any time.

Yeah I see the results, and it is so challenging for the model lmao. There is a lot of room to improve this model. Thanks for your effort and interest.

[phr00t]

Here you can see I am using a "frame stride" of 10, which was the default value. I don't really know how that parameter works, but it sounds like I'd still need to infer 16 frames with different frame stride values, so I wouldn't save any time, correct?

Larger fs normally means the generated motion magnitude could be smaller (more stable results), and vice versa. Yeah you need to infer 16 frames anyway (but more frames are possible I believe?) and cannot save any time.

Yeah I see the results, and it is so challenging for the model lmao. There is a lot of room to improve this model. Thanks for your effort and interest.

I didn't know where else to post this, but I've made significant progress here and I've gotten it working quite well with very different keyframes at only 9 steps! This was with 3 frames (sitting, standing and pointing):

https://github.com/Doubiiu/DynamiCrafter/assets/5983470/83c0d253-4dce-446f-9c42-729910242478

[jryebread]

@phr00t this looks great can you share what settings?

[phr00t]

@phr00t this looks great can you share what settings?

Look for more information here: https://www.reddit.com/r/StableDiffusion/comments/1bia33a/comment/kvj0d47/