search

sign-language-processing / pose-to-video

Render pose sequences as photorealistic videos.
8 stars 6 forks source link

Integrate latest AnimateDiff-Lightning model #3

Open hensuninnovation opened 5 months ago

hensuninnovation commented 5 months ago

Hello! Thank you very much for open source these wonderful sign language processing toolkit and models! Truly remarkable! I come across this new AnimateDiff model from ByteDance, which looks fast and promising. I am wondering how hard to integrate into this project.

https://huggingface.co/ByteDance/AnimateDiff-Lightning

https://arxiv.org/pdf/2403.12706.pdf

AmitMY commented 5 months ago

It should be as simple as changing the adapter ID, scheduler config, and maybe adjusting some of the diffusion parameters.

If you have access to a GPU, you could play with it immediately here: https://github.com/sign-language-processing/pose-to-video/blob/main/pose_to_video/processors/animatediff/__init__.py

I don't expect it to take too much work. Feel free to contribute a solution! :)