open-mmlab / PIA

[CVPR 2024] PIA, your Personalized Image Animator. Animate your images by text prompt, combing with Dreambooth, achieving stunning videos. PIA,你的个性化图像动画生成器,利用文本提示将图像变为奇妙的动画
https://pi-animator.github.io/
Apache License 2.0
888 stars 70 forks source link

Can PIA checkpoint be used in A1111 webui? #18

Closed xpeng closed 9 months ago

xpeng commented 9 months ago

great work! can we put PIA checkpoint into animatediff webui extention as some motion models or else?

LeoXing1996 commented 9 months ago

@xpeng Unfortunately, the animatediff extention cannot load PIA right now. You can try requesting this feature in the corresponding extension's repository.

johndpope commented 9 months ago

Hello @LeoXing1996 - I got this working and now very keen to try with my other sd models. (i did stumble on just selecting a lora from list - and it was showing the models - and thus breaks things - that second list should be restricted to lora only)

given that this code seems to extend animatediff - can we just hot swap (take included directory) and replace the base animatediff-webui codebase and it should just work? https://github.com/continue-revolution/sd-webui-animatediff does heart of logic lie in side that? was interface changed? have to dig around. fyi @continue-revolution

continue-revolution commented 9 months ago

If someone can explain the difference this repo made to animatediff (from code level) then it is easy for me to support that, otherwise I have to study the source code and it’s gonna be too time consuming.

johndpope commented 9 months ago

@LeoXing1996 - is is possible to update your animatediff as much as possible to bring in line with upstream? I paste your code into upstream and it seems like there's some regression https://github.com/johndpope/AnimateDiff/pull/1/files InflatedGroupNorm / use_inflated_groupnorm

animatediff/utils/util.py

animatediff/utils/convert_lora_safetensor_to_diffusers.py

animatediff/utils/convert_from_ckpt.py

is there a complication with PIA where it can't support controlnet?