search

showlab / MotionDirector

[ECCV 2024 Oral] MotionDirector: Motion Customization of Text-to-Video Diffusion Models.
https://showlab.github.io/MotionDirector/
Apache License 2.0
850 stars 54 forks source link

Why can't I reproduce the beautiful results shown by the author even using the weight file provided by the author on huggingface #37

Closed LiuHuijie6410 closed 3 months ago

LiuHuijie6410 commented 6 months ago

Congratulations on such a beautiful job！ However, I have always been unable to reproduce the exquisite results shown by the author. Have I done something wrong somewhere? For example, I use the lora weights from huggingface provided by the author(e.g. golf https://huggingface.co/ruizhaocv/MotionDirector/tree/main/playing_golf) Then I use the same random seed provided by the author for inference， python MotionDirector_inference.py --model "models/zeroscope_v2_576w" --prompt "A monkey is playing golf on a field full of flowers." --checkpoint_folder /MotionDirector/huggingface/playing_golf/ --checkpoint_index 300 --noise_prior 0. --seed 2989633 I get the output, A_monkey_is_playing_golf_on_a_field_full_of_flowers_2989633

The video shown by the author is A_monkey_is_playing_golf_on_a_field_full_of_flowers_2989633

May I ask what I have done wrong that may prevent me from reproducing the effect presented by the author?

ruizhaocv commented 3 months ago

Sorry for the late reply. We developed two versions of checkpoints, including one mentioned in the github repo, and another one used for huggingface space, thus the provided random seed may be inconsistent with the huggingface space weights. Another possible reason is the different torch version causing the generated random noises with same seed inconsistent.