open-mmlab / PIA

[CVPR 2024] PIA, your Personalized Image Animator. Animate your images by text prompt, combing with Dreambooth, achieving stunning videos. PIA,你的个性化图像动画生成器,利用文本提示将图像变为奇妙的动画
https://pi-animator.github.io/
Apache License 2.0
913 stars 75 forks source link

Can PIA checkpoint be used in A1111 webui? #18

Closed xpeng closed 10 months ago

xpeng commented 10 months ago

great work! can we put PIA checkpoint into animatediff webui extention as some motion models or else?

LeoXing1996 commented 10 months ago

@xpeng Unfortunately, the animatediff extention cannot load PIA right now. You can try requesting this feature in the corresponding extension's repository.

johndpope commented 10 months ago

Hello @LeoXing1996 - I got this working and now very keen to try with my other sd models. (i did stumble on just selecting a lora from list - and it was showing the models - and thus breaks things - that second list should be restricted to lora only)

given that this code seems to extend animatediff - can we just hot swap (take included directory) and replace the base animatediff-webui codebase and it should just work? https://github.com/continue-revolution/sd-webui-animatediff does heart of logic lie in side that? was interface changed? have to dig around. fyi @continue-revolution

continue-revolution commented 10 months ago

If someone can explain the difference this repo made to animatediff (from code level) then it is easy for me to support that, otherwise I have to study the source code and it’s gonna be too time consuming.

johndpope commented 10 months ago

@LeoXing1996 - is is possible to update your animatediff as much as possible to bring in line with upstream? I paste your code into upstream and it seems like there's some regression https://github.com/johndpope/AnimateDiff/pull/1/files InflatedGroupNorm / use_inflated_groupnorm

animatediff/utils/util.py

animatediff/utils/convert_lora_safetensor_to_diffusers.py

animatediff/utils/convert_from_ckpt.py

is there a complication with PIA where it can't support controlnet?