search

Doubiiu / DynamiCrafter

[ECCV 2024, Oral] DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
https://doubiiu.github.io/projects/DynamiCrafter/
Apache License 2.0
2.46k stars 197 forks source link

Arbitrary resolution #22

Open HyoKong opened 7 months ago

HyoKong commented 7 months ago

Hi, thank you so much for your great work!

May I ask how to adapt to images with arbitrary resolutions, for instance, 512x768?

Thank you so much for your help!

Doubiiu commented 7 months ago

Hi Thanks for your interest! I am not sure how the model performs on the arbitrary resolution inputs. You can directly try it using both 512 model and 1024 model, and modify the input resolution in run.sh or run_mp.sh, i.e. --height ${H} --width $1 to the required height and width. I think the model still works to some extent in this way.

HyoKong commented 7 months ago

Hi, adjusting height may be not working. there are bugs when concatenating the output of downsampling and upsampling blocks. The shapes do not match.

https://github.com/Doubiiu/DynamiCrafter/blob/ce86a1207494e11f7befda7bfb5235ea8f9d518e/lvdm/modules/networks/openaimodel3d.py#L596

Could you pls help to solve this issue?

Doubiiu commented 7 months ago

Hi, what is your exact resolution for inference? To my best knowledge, the height and width should be divided by 32 (I think this is the reason of your bug). And can you summarize the change of code for the target resolution inference?