How to use MotionDirector to train a LoRA based on other base models?

showlab / MotionDirector

[ECCV 2024 Oral] MotionDirector: Motion Customization of Text-to-Video Diffusion Models.

https://showlab.github.io/MotionDirector/

Apache License 2.0

814 stars 48 forks source link

How to use MotionDirector to train a LoRA based on other base models? #43

Closed LiuHuijie6410 closed 1 month ago

LiuHuijie6410 commented 1 month ago

Thanks for such amazing work！ I want to use MotionDirector to train LoRa based on Stable Video Diffusion or ModelScope_v1.5. Is this easy to implement? Do I need to rewrite the code in utils to insert lora? ps. The ModelScope_v1.5 I mentioned is https://modelscope.cn/models/iic/dreamvideo-t2v/summary. Thanks for your reply!

ruizhaocv commented 1 month ago

Yes, you need to rewrite to adapt to their code base.