add animatediff training pipeline - Githubissues

mindspore-lab / mindone

one for all, Optimal generator with No Exception

https://mindspore-lab.github.io/mindone/

Apache License 2.0

338 stars 63 forks source link

add animatediff training pipeline #311

Closed SamitHuang closed 5 months ago

SamitHuang commented 5 months ago

mainly tested on GPU and 910B, training results are as expected for input video data in 256x256, frames=16.

Features added:

[x] motion module v1&v2 training
[x] motion lora training
[x] common utilities for training like lr scheduler are extracted to mindone lib
[x] lora injection are extracted to mindone lib

TODOs:

[ ] Support V3, SDXL
[ ] Support distributed training in 512x512 in high performance
[ ] Precision config optimization
[ ] Improve temporal consistency and spatial fidelity