search

jianzongwu / MotionBooth

The official implement of research paper "MotionBooth: Motion-Aware Customized Text-to-Video Generation"
72 stars 5 forks source link

LaVie not working #5

Closed Junoh-Kang closed 2 weeks ago

Junoh-Kang commented 2 weeks ago

Dear authors,

I have problem with inferencing using LaVie model. I followed the instructions, but I encountered the following errors. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/sensei-fs/users/junohk/MotionBooth/scripts/inference.py", line 147, in <module> main(args) File "/sensei-fs/users/junohk/MotionBooth/scripts/inference.py", line 46, in main unet = model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/motionbooth/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/motionbooth/lib/python3.11/site-packages/diffusers/models/modeling_utils.py", line 740, in from_pretrained model = cls.from_config(config, **unused_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/motionbooth/lib/python3.11/site-packages/diffusers/configuration_utils.py", line 260, in from_config model = cls(**init_dict) ^^^^^^^^^^^^^^^^ File "/opt/conda/envs/motionbooth/lib/python3.11/site-packages/diffusers/configuration_utils.py", line 658, in inner_init init(self, *args, **init_kwargs) File "/sensei-fs/users/junohk/MotionBooth/src/models/lavie/unet.py", line 180, in __init__ down_block = get_down_block( ^^^^^^^^^^^^^^^ File "/sensei-fs/users/junohk/MotionBooth/src/models/lavie/unet_blocks.py", line 75, in get_down_block raise ValueError(f"{down_block_type} does not exist.") ValueError: CrossAttnDownBlock2D does not exist.

I found that the codes will not raise error if the down_block_type is CrossAttnDownBlock3D, but I have only 2Ds. (Pdb) down_block_types ['CrossAttnDownBlock2D', 'CrossAttnDownBlock2D', 'CrossAttnDownBlock2D', 'DownBlock2D']

jianzongwu commented 2 weeks ago

Hello, please see this issue https://github.com/jianzongwu/MotionBooth/issues/2

Apologize that I missed some LaVie checkpoint convert steps in the first version of README.md.

Now I have added them. Please check README.md again and do the new steps.

Concretely, it is about editing some config files in the converted LaVie checkpoint

Junoh-Kang commented 2 weeks ago

Thank you for a quick answer!