search

ali-vilab / UniAnimate

Code for Paper "UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation".
https://unianimate.github.io/
565 stars 35 forks source link

Can't generate video #46

Closed zhenyuanzhou closed 4 days ago

zhenyuanzhou commented 5 days ago

Traceback (most recent call last): File "d:\PythonProject\UniAnimate-GradioUI\inference.py", line 18, in INFER_ENGINE.build(dict(type=cfg_update.TASK_TYPE), cfg_update=cfg_update.cfg_dict) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 107, in build return self.build_func(*args, kwargs, registry=self) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry_class.py", line 7, in build_func return build_from_config(cfg, registry, kwargs) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 55, in build_from_config raise KeyError(f"{req_type} not found in {registry.name} registry") KeyError: 'inference_unianimate_entrance not found in INFER_ENGINE registry'

nitinmukesh commented 5 days ago

@zhenyuanzhou

It seems you managed to install on Windows. Please if you can share results of pip list

I am struggling with this issue https://github.com/ali-vilab/UniAnimate/issues/43

zhenyuanzhou commented 5 days ago

(D:\PythonProject\UniAnimate-GradioUI\venv) D:\PythonProject\UniAnimate-GradioUI>pip list Package Version

aiofiles 23.2.1 aiohttp 3.9.5 aiosignal 1.3.1 aliyun-python-sdk-core 2.15.1 aliyun-python-sdk-kms 2.16.3 altair 5.3.0 annotated-types 0.7.0 ansicon 1.89.0 antlr4-python3-runtime 4.9.3 anyio 4.4.0 artist 0.18.2 asttokens 2.4.1 async-timeout 4.0.3 attrs 23.2.0 Automat 22.10.0 beartype 0.18.5 blessed 1.20.0 buildtools 1.0.6 certifi 2024.7.4 cffi 1.16.0 chardet 5.2.0 charset-normalizer 3.3.2 clean-fid 0.1.35 click 8.1.7 clip 0.2.0 cmake 3.30.0 colorama 0.4.6 coloredlogs 15.0.1 comm 0.2.2 constantly 23.10.4 contourpy 1.2.1 crcmod 1.7 cryptography 42.0.8 cycler 0.12.1 debugpy 1.8.2 decorator 5.1.1 decord 0.6.0 diffusers 0.29.2 dnspython 2.6.1 docopt 0.6.2 easydict 1.13 einops 0.8.0 email_validator 2.2.0 exceptiongroup 1.2.0 executing 2.0.1 fairscale 0.4.13 fastapi 0.111.0 fastapi-cli 0.0.4 ffmpeg 1.4 ffmpy 0.3.2 filelock 3.15.4 flatbuffers 24.3.25 fonttools 4.53.0 frozenlist 1.4.1 fsspec 2024.6.1 ftfy 6.2.0 furl 2.1.3 gpustat 1.1.1 gradio 4.37.2 gradio_client 1.0.2 greenlet 3.0.3 h11 0.14.0 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.23.4 humanfriendly 10.0 hyperlink 21.0.0 idna 3.7 imageio 2.34.2 imageio-ffmpeg 0.5.1 importlib_metadata 8.0.0 importlib_resources 6.4.0 incremental 22.10.0 ipdb 0.13.13 ipykernel 6.29.5 ipython 8.26.0 jedi 0.19.1 Jinja2 3.1.4 jinxed 1.2.1 jmespath 0.10.0 joblib 1.4.2 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 jupyter_client 8.6.2 jupyter_core 5.7.2 kiwisolver 1.4.5 kornia 0.7.3 kornia_rs 0.1.4 lazy_loader 0.4 lightning-utilities 0.11.3.post0 lit 18.1.8 lpips 0.1.4 lxml 5.2.2 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.9.0 matplotlib-inline 0.1.7 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.5 mypy-extensions 1.0.0 nest_asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 numpy 1.26.4 nvidia-cublas-cu12 12.5.3.2 nvidia-cuda-cupti-cu12 12.5.82 nvidia-cuda-nvrtc-cu12 12.5.82 nvidia-cuda-runtime-cu12 12.5.82 nvidia-cudnn-cu12 9.2.0.82 nvidia-cufft-cu12 11.2.3.61 nvidia-curand-cu12 10.3.6.82 nvidia-cusolver-cu12 11.6.3.83 nvidia-cusparse-cu12 12.5.1.3 nvidia-ml-py 12.555.43 nvidia-nvjitlink-cu12 12.5.82 nvidia-nvtx-cu12 12.5.82 omegaconf 2.3.0 onnxruntime-gpu 1.13.1 open-clip-torch 2.24.0 opencv-python-headless 4.9.0.80 orderedmultidict 1.0.1 orjson 3.10.6 oss2 2.18.6 packaging 24.1 pandas 2.2.2 parso 0.8.4 pexpect 4.9.0 pickleshare 0.7.5 pillow 10.4.0 pip 24.0 piq 0.8.0 pkgconfig 1.5.5 platformdirs 4.2.2 prompt_toolkit 3.0.47 protobuf 5.27.2 psutil 6.0.0 ptflops 0.7.3 ptyprocess 0.7.0 pure-eval 0.2.2 pycparser 2.22 pycryptodome 3.20.0 pydantic 2.8.1 pydantic_core 2.20.1 pydub 0.25.1 Pygments 2.18.0 pynvml 11.5.0 pyparsing 3.1.2 pyre-extensions 0.0.30 pyreadline3 3.4.1 python-dateutil 2.9.0 python-dotenv 1.0.1 python-multipart 0.0.9 pytils 0.4.1 pytorch-lightning 2.3.1 pytz 2024.1 pywin32 306 PyYAML 6.0.1 pyzmq 26.0.3 redo 2.0.4 referencing 0.35.1 regex 2024.5.15 requests 2.32.3 rich 13.7.1 rotary-embedding-torch 0.6.4 rpds-py 0.18.1 ruff 0.5.0 safetensors 0.4.3 scikit-image 0.24.0 scikit-learn 1.5.1 scipy 1.14.0 semantic-version 2.10.0 sentencepiece 0.2.0 setuptools 69.5.1 shellingham 1.5.4 simplejson 3.19.2 six 1.16.0 sk-video 1.1.10 sniffio 1.3.1 SQLAlchemy 2.0.31 stack-data 0.6.2 starlette 0.37.2 sympy 1.12.1 thop 0.1.1.post2209072238 threadpoolctl 3.5.0 tifffile 2024.7.2 timm 1.0.7 tokenizers 0.19.1 tomli 2.0.1 tomlkit 0.12.0 toolz 0.12.1 torch 2.2.2+cu121 torchdiffeq 0.2.4 torchmetrics 1.4.0.post0 torchsde 0.2.6 torchvision 0.17.2 tornado 6.4.1 tqdm 4.66.4 traitlets 5.14.3 trampoline 0.1.2 transformers 4.42.3 Twisted 24.3.0 twisted-iocpsupport 1.0.4 typer 0.12.3 typing_extensions 4.12.2 typing-inspect 0.9.0 tzdata 2024.1 ujson 5.10.0 urllib3 2.2.2 uvicorn 0.30.1 watchfiles 0.22.0 wcwidth 0.2.13 websockets 11.0.3 wheel 0.43.0 xformers 0.0.25.post1 yarl 1.9.4 zipp 3.19.2 zope.interface 6.4.post2

Thank you!

wangxiang1230 commented 5 days ago

Traceback (most recent call last): File "d:\PythonProject\UniAnimate-GradioUI\inference.py", line 18, in INFER_ENGINE.build(dict(type=cfg_update.TASK_TYPE), cfg_update=cfg_update.cfg_dict) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 107, in build return self.build_func(*args, kwargs, registry=self) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry_class.py", line 7, in build_func return build_from_config(cfg, registry, kwargs) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 55, in build_from_config raise KeyError(f"{req_type} not found in {registry.name} registry") KeyError: 'inference_unianimate_entrance not found in INFER_ENGINE registry'

Hi, maybe you can try to explicitly replace https://github.com/ali-vilab/UniAnimate/blob/549ee5fad7618500790929b0ae73151d36649045/tools/inferences/inference_unianimate_entrance.py#L55 to @INFER_ENGINE.register_function("inference_unianimate_entrance")

zhenyuanzhou commented 4 days ago

Traceback (most recent call last): File "d:\PythonProject\UniAnimate-GradioUI\inference.py", line 18, in INFER_ENGINE.build(dict(type=cfg_update.TASK_TYPE), cfg_update=cfg_update.cfg_dict) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 107, in build return self.build_func(*args, kwargs, registry=self) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry_class.py", line 7, in build_func return build_from_config(cfg, registry, kwargs) File "d:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 55, in build_from_config raise KeyError(f"{req_type} not found in {registry.name} registry") KeyError: 'inference_unianimate_entrance not found in INFER_ENGINE registry'

Hi, maybe you can try to explicitly replace

https://github.com/ali-vilab/UniAnimate/blob/549ee5fad7618500790929b0ae73151d36649045/tools/inferences/inference_unianimate_entrance.py#L55

to @INFER_ENGINE.register_function("inference_unianimate_entrance")

Thank you, but don't work for me. 屏幕截图 2024-07-04 215944 (D:\PythonProject\UniAnimate-GradioUI\venv) D:\PythonProject\UniAnimate-GradioUI>python inference.py --cfg configs/UniAnimate_infer.yaml Traceback (most recent call last): File "D:\PythonProject\UniAnimate-GradioUI\inference.py", line 18, in INFER_ENGINE.build(dict(type=cfg_update.TASK_TYPE), cfg_update=cfg_update.cfg_dict) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 107, in build return self.build_func(*args, kwargs, registry=self) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry_class.py", line 7, in build_func return build_from_config(cfg, registry, kwargs) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 55, in build_from_config raise KeyError(f"{req_type} not found in {registry.name} registry") KeyError: 'inference_unianimate_entrance not found in INFER_ENGINE registry'

Any other methods? Thank you!

zhenyuanzhou commented 4 days ago

屏幕截图 2024-07-04 215930

zhenyuanzhou commented 4 days ago

屏幕截图 2024-07-04 221334

zhenyuanzhou commented 4 days ago

Another problem is I need to be able to use CUDA to generate poses. But this is fine; I can use CPU instead. The weird thing is with all the settings are correct, the programs standstill.

wangxiang1230 commented 4 days ago

Another problem is I need to be able to use CUDA to generate poses. But this is fine; I can use CPU instead. The weird thing is with all the settings are correct, the programs standstill.

Hi, do you use our latest version? The pose extraction can be run on CUDA. We have fixed it (refer to https://github.com/ali-vilab/UniAnimate/issues/29)

wangxiang1230 commented 4 days ago

Hi, for the first question. You can change https://github.com/ali-vilab/UniAnimate/blob/549ee5fad7618500790929b0ae73151d36649045/inference.py#L11-L18 to

from utils.config import Config 
from utils.registry_class import INFER_ENGINE 

from tools import * 
from tools.inferences.inference_unianimate_entrance import inference_unianimate_entrance

if __name__ == '__main__':
    cfg_update = Config(load=True)
    inference_unianimate_entrance(cfg_update=cfg_update.cfg_dict)
    # INFER_ENGINE.build(dict(type=cfg_update.TASK_TYPE), cfg_update=cfg_update.cfg_dict)

Hope that helps you, thanks.

zhenyuanzhou commented 4 days ago

Thank you. The new problem is emergent. Because Windows does not support it, I replaced the NCCL with gloo. Then it works by adding these two codes into inference_unianimate_entrance.py and replace the NCCL with gloo. import os os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

Now the problem is below, I believe this is similar to the previous one. Is that anyway to solve all these importation problem once for all? Thank you.

(D:\PythonProject\UniAnimate-GradioUI\venv) D:\PythonProject\UniAnimate-GradioUI>python inference.py --cfg configs/UniAnimate_infer.yaml [2024-07-04 22:46:19,751] INFO: {'name': 'Config: VideoLDM Decoder', 'mean': [0.5, 0.5, 0.5], 'std': [0.5, 0.5, 0.5], 'max_words': 1000, 'num_workers': 8, 'prefetch_factor': 2, 'resolution': [512, 768], 'vit_out_dim': 1024, 'vit_resolution': [224, 224], 'depth_clamp': 10.0, 'misc_size': 384, 'depth_std': 20.0, 'save_fps': 8, 'frame_lens': [32, 32, 32, 1], 'sample_fps': [4], 'vid_dataset': {'type': 'VideoBaseDataset', 'data_list': [], 'max_words': 1000, 'resolution': [448, 256]}, 'img_dataset': {'type': 'ImageBaseDataset', 'data_list': ['laion_400m'], 'max_words': 1000, 'resolution': [448, 256]}, 'batch_sizes': {'1': 256, '4': 4, '8': 4, '16': 4}, 'Diffusion': {'type': 'DiffusionDDIM', 'schedule': 'linear_sd', 'schedule_param': {'init_beta': 0.00085, 'last_beta': 0.012, 'num_timesteps': 1000, 'zero_terminal_snr': True}, 'mean_type': 'v', 'loss_type': 'mse', 'var_type': 'fixed_small', 'rescale_timesteps': False, 'noise_strength': 0.1, 'ddim_timesteps': 50}, 'ddim_timesteps': 30, 'use_div_loss': False, 'p_zero': 0.9, 'guide_scale': 2.5, 'vit_mean': [0.48145466, 0.4578275, 0.40821073], 'vit_std': [0.26862954, 0.26130258, 0.27577711], 'sketch_mean': [0.485, 0.456, 0.406], 'sketch_std': [0.229, 0.224, 0.225], 'hist_sigma': 10.0, 'scale_factor': 0.18215, 'use_checkpoint': True, 'use_sharded_ddp': False, 'use_fsdp': False, 'use_fp16': True, 'temporal_attention': True, 'UNet': {'type': 'UNetSD_UniAnimate', 'in_dim': 4, 'dim': 320, 'y_dim': 1024, 'context_dim': 1024, 'out_dim': 4, 'dim_mult': [1, 2, 4, 4], 'num_heads': 8, 'head_dim': 64, 'num_res_blocks': 2, 'attn_scales': [1.0, 0.5, 0.25], 'dropout': 0.1, 'temporal_attention': True, 'temporal_attn_times': 1, 'use_checkpoint': True, 'use_fps_condition': False, 'use_sim_mask': False, 'config': 'None', 'num_tokens': 4}, 'guidances': [], 'auto_encoder': {'type': 'AutoencoderKL', 'ddconfig': {'attn_resolutions': [], 'ch': 128, 'ch_mult': [1, 2, 4, 4], 'double_z': True, 'dropout': 0.0, 'in_channels': 3, 'num_res_blocks': 2, 'out_ch': 3, 'resolution': 256, 'video_kernel_size': [3, 1, 1], 'z_channels': 4}, 'embed_dim': 4, 'pretrained': 'checkpoints/v2-1_512-ema-pruned.ckpt'}, 'embedder': {'type': 'FrozenOpenCLIPTextVisualEmbedder', 'layer': 'penultimate', 'pretrained': 'checkpoints/open_clip_pytorch_model.bin'}, 'ema_decay': 0.9999, 'num_steps': 600000, 'lr': 5e-05, 'weight_decay': 0.0, 'betas': (0.9, 0.999), 'eps': 1e-08, 'chunk_size': 2, 'decoder_bs': 2, 'alpha': 0.7, 'save_ckp_interval': 1000, 'warmup_steps': 10, 'decay_mode': 'cosine', 'use_ema': False, 'load_from': None, 'Pretrain': {'type': 'pretrain_specific_strategies', 'fix_weight': False, 'grad_scale': 0.2, 'resume_checkpoint': 'models/jiuniu_0267000.pth', 'sd_keys_path': 'models/stable_diffusion_image_key_temporal_attention_x1.json'}, 'viz_interval': 1000, 'visual_train': {'type': 'VisualTrainTextImageToVideo'}, 'visual_inference': {'type': 'VisualGeneratedVideos'}, 'inference_list_path': '', 'log_interval': 100, 'log_dir': 'outputs/UniAnimate_infer', 'seed': 18, 'negative_prompt': 'Distorted, discontinuous, Ugly, blurry, low resolution, motionless, static, disfigured, disconnected limbs, Ugly faces, incomplete arms', 'CPU_CLIP_VAE': True, 'TASK_TYPE': 'inference_unianimate_entrance', 'batch_size': 1, 'latent_random_ref': True, 'max_frames': 32, 'partial_keys': [['image', 'local_image', 'dwpose'], ['image', 'randomref', 'dwpose']], 'round': 1, 'scale': 8, 'test_list_path': [[2, 'data/images/musk.jpg', 'data/saved_pose/source_video/automan']], 'test_model': 'checkpoints/unianimate_16f_32f_non_ema_223000.pth', 'use_DiffusionDPM': False, 'use_fps_condition': False, 'video_compositions': ['image', 'local_image', 'dwpose', 'randomref', 'randomref_pose'], 'cfg_file': 'configs/UniAnimate_infer.yaml', 'init_method': 'tcp://localhost:9999', 'debug': False, 'opts': [], 'pmi_rank': 0, 'pmi_world_size': 1, 'gpus_per_machine': 1, 'world_size': 1, 'gpu': 0, 'rank': 0, 'log_file': 'outputs/UniAnimate_infer\log_00.txt'} [2024-07-04 22:46:19,752] INFO: Running UniAnimate inference on gpu 0 Traceback (most recent call last): File "D:\PythonProject\UniAnimate-GradioUI\inference.py", line 28, in inference_unianimate_entrance(cfg_update=cfg_update.cfg_dict) File "D:\PythonProject\UniAnimate-GradioUI\tools\inferences\inference_unianimate_entrance.py", line 81, in inference_unianimate_entrance worker(0, cfg, cfg_update) File "D:\PythonProject\UniAnimate-GradioUI\tools\inferences\inference_unianimate_entrance.py", line 233, in worker diffusion = DIFFUSION.build(cfg.Diffusion) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 107, in build return self.build_func(*args, kwargs, registry=self) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry_class.py", line 7, in build_func return build_from_config(cfg, registry, kwargs) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 55, in build_from_config raise KeyError(f"{req_type} not found in {registry.name} registry") KeyError: 'DiffusionDDIM not found in DIFFUSION registry'

wangxiang1230 commented 4 days ago

It's a bad news that all your Registry_class are not working... As every time, you need to similar to https://github.com/ali-vilab/UniAnimate/issues/46#issuecomment-2209138104 explicitly call functions...

wangxiang1230 commented 4 days ago

Hi, you can try to change https://github.com/ali-vilab/UniAnimate/blob/549ee5fad7618500790929b0ae73151d36649045/utils/registry.py#L50-L53

to

req_type = cfg.pop("type") 
req_type_entry = req_type 
if isinstance(req_type, str): 
    from tools.inferences.inference_unianimate_entrance import inference_unianimate_entrance
    from tools.modules.diffusions.diffusion_ddim import DiffusionDDIM
    from tools.modules.autoencoder import AutoencoderKL
    from tools.modules.clip_embedder import FrozenOpenCLIPTextVisualEmbedder
    from tools.modules.unet.unet_unianimate import UNetSD_UniAnimate

    req_type_entry = eval(req_type)
    # req_type_entry = registry.get(req_type)

If you meet similar cases, try to import it explicitly. Hope this helps you.

zhenyuanzhou commented 4 days ago

It's a bad news that all your Registry_class are not working... As every time, you need to similar to #46 (comment) explicitly call functions...

Sorry to bother you again, but I'm an amateur player and never learned how to program. I added the following code into inference.py, but still got the same problem. Any thing I can do to solve it?

from utils.config import Config from utils.registry_class import INFER_ENGINE from utils.registry_class import DIFFUSION from utils.registry_class import AUTO_ENCODER from utils.registry_class import DATASETS from utils.registry_class import DISTRIBUTION from utils.registry_class import EMBEDDER from utils.registry_class import ENGINE from utils.registry_class import MODEL from utils.registry_class import PRETRAIN from utils.registry_class import VISUAL from utils.registry_class import EMBEDMANAGER

zhenyuanzhou commented 4 days ago

屏幕截图 2024-07-04 235247 The code is like this

zhenyuanzhou commented 4 days ago

(D:\PythonProject\UniAnimate-GradioUI\venv) D:\PythonProject\UniAnimate-GradioUI>python inference.py --cfg configs/UniAnimate_infer.yaml [2024-07-04 23:53:31,932] INFO: {'name': 'Config: VideoLDM Decoder', 'mean': [0.5, 0.5, 0.5], 'std': [0.5, 0.5, 0.5], 'max_words': 1000, 'num_workers': 8, 'prefetch_factor': 2, 'resolution': [512, 768], 'vit_out_dim': 1024, 'vit_resolution': [224, 224], 'depth_clamp': 10.0, 'misc_size': 384, 'depth_std': 20.0, 'save_fps': 8, 'frame_lens': [32, 32, 32, 1], 'sample_fps': [4], 'vid_dataset': {'type': 'VideoBaseDataset', 'data_list': [], 'max_words': 1000, 'resolution': [448, 256]}, 'img_dataset': {'type': 'ImageBaseDataset', 'data_list': ['laion_400m'], 'max_words': 1000, 'resolution': [448, 256]}, 'batch_sizes': {'1': 256, '4': 4, '8': 4, '16': 4}, 'Diffusion': {'type': 'DiffusionDDIM', 'schedule': 'linear_sd', 'schedule_param': {'init_beta': 0.00085, 'last_beta': 0.012, 'num_timesteps': 1000, 'zero_terminal_snr': True}, 'mean_type': 'v', 'loss_type': 'mse', 'var_type': 'fixed_small', 'rescale_timesteps': False, 'noise_strength': 0.1, 'ddim_timesteps': 50}, 'ddim_timesteps': 30, 'use_div_loss': False, 'p_zero': 0.9, 'guide_scale': 2.5, 'vit_mean': [0.48145466, 0.4578275, 0.40821073], 'vit_std': [0.26862954, 0.26130258, 0.27577711], 'sketch_mean': [0.485, 0.456, 0.406], 'sketch_std': [0.229, 0.224, 0.225], 'hist_sigma': 10.0, 'scale_factor': 0.18215, 'use_checkpoint': True, 'use_sharded_ddp': False, 'use_fsdp': False, 'use_fp16': True, 'temporal_attention': True, 'UNet': {'type': 'UNetSD_UniAnimate', 'in_dim': 4, 'dim': 320, 'y_dim': 1024, 'context_dim': 1024, 'out_dim': 4, 'dim_mult': [1, 2, 4, 4], 'num_heads': 8, 'head_dim': 64, 'num_res_blocks': 2, 'attn_scales': [1.0, 0.5, 0.25], 'dropout': 0.1, 'temporal_attention': True, 'temporal_attn_times': 1, 'use_checkpoint': True, 'use_fps_condition': False, 'use_sim_mask': False, 'config': 'None', 'num_tokens': 4}, 'guidances': [], 'auto_encoder': {'type': 'AutoencoderKL', 'ddconfig': {'attn_resolutions': [], 'ch': 128, 'ch_mult': [1, 2, 4, 4], 'double_z': True, 'dropout': 0.0, 'in_channels': 3, 'num_res_blocks': 2, 'out_ch': 3, 'resolution': 256, 'video_kernel_size': [3, 1, 1], 'z_channels': 4}, 'embed_dim': 4, 'pretrained': 'checkpoints/v2-1_512-ema-pruned.ckpt'}, 'embedder': {'type': 'FrozenOpenCLIPTextVisualEmbedder', 'layer': 'penultimate', 'pretrained': 'checkpoints/open_clip_pytorch_model.bin'}, 'ema_decay': 0.9999, 'num_steps': 600000, 'lr': 5e-05, 'weight_decay': 0.0, 'betas': (0.9, 0.999), 'eps': 1e-08, 'chunk_size': 2, 'decoder_bs': 2, 'alpha': 0.7, 'save_ckp_interval': 1000, 'warmup_steps': 10, 'decay_mode': 'cosine', 'use_ema': False, 'load_from': None, 'Pretrain': {'type': 'pretrain_specific_strategies', 'fix_weight': False, 'grad_scale': 0.2, 'resume_checkpoint': 'models/jiuniu_0267000.pth', 'sd_keys_path': 'models/stable_diffusion_image_key_temporal_attention_x1.json'}, 'viz_interval': 1000, 'visual_train': {'type': 'VisualTrainTextImageToVideo'}, 'visual_inference': {'type': 'VisualGeneratedVideos'}, 'inference_list_path': '', 'log_interval': 100, 'log_dir': 'outputs/UniAnimate_infer', 'seed': 18, 'negative_prompt': 'Distorted, discontinuous, Ugly, blurry, low resolution, motionless, static, disfigured, disconnected limbs, Ugly faces, incomplete arms', 'CPU_CLIP_VAE': True, 'TASK_TYPE': 'inference_unianimate_entrance', 'batch_size': 1, 'latent_random_ref': True, 'max_frames': 32, 'partial_keys': [['image', 'local_image', 'dwpose'], ['image', 'randomref', 'dwpose']], 'round': 1, 'scale': 8, 'test_list_path': [[2, 'data/images/musk.jpg', 'data/saved_pose/source_video/automan']], 'test_model': 'checkpoints/unianimate_16f_32f_non_ema_223000.pth', 'use_DiffusionDPM': False, 'use_fps_condition': False, 'video_compositions': ['image', 'local_image', 'dwpose', 'randomref', 'randomref_pose'], 'cfg_file': 'configs/UniAnimate_infer.yaml', 'init_method': 'tcp://localhost:9999', 'debug': False, 'opts': [], 'pmi_rank': 0, 'pmi_world_size': 1, 'gpus_per_machine': 1, 'world_size': 1, 'gpu': 0, 'rank': 0, 'log_file': 'outputs/UniAnimate_infer\log_00.txt'} [2024-07-04 23:53:31,932] INFO: Running UniAnimate inference on gpu 0 Traceback (most recent call last): File "D:\PythonProject\UniAnimate-GradioUI\inference.py", line 41, in inference_unianimate_entrance(cfg_update=cfg_update.cfg_dict) File "D:\PythonProject\UniAnimate-GradioUI\tools\inferences\inference_unianimate_entrance.py", line 81, in inference_unianimate_entrance worker(0, cfg, cfg_update) File "D:\PythonProject\UniAnimate-GradioUI\tools\inferences\inference_unianimate_entrance.py", line 233, in worker diffusion = DIFFUSION.build(cfg.Diffusion) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 107, in build return self.build_func(*args, kwargs, registry=self) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry_class.py", line 7, in build_func return build_from_config(cfg, registry, kwargs) File "D:\PythonProject\UniAnimate-GradioUI\utils\registry.py", line 55, in build_from_config raise KeyError(f"{req_type} not found in {registry.name} registry") KeyError: 'DiffusionDDIM not found in DIFFUSION registry'

wangxiang1230 commented 4 days ago

Hi, you can try to change

https://github.com/ali-vilab/UniAnimate/blob/549ee5fad7618500790929b0ae73151d36649045/utils/registry.py#L50-L53

to

req_type = cfg.pop("type") 
req_type_entry = req_type 
if isinstance(req_type, str): 
    from tools.inferences.inference_unianimate_entrance import inference_unianimate_entrance
    from tools.modules.diffusions.diffusion_ddim import DiffusionDDIM
    from tools.modules.autoencoder import AutoencoderKL
    from tools.modules.clip_embedder import FrozenOpenCLIPTextVisualEmbedder
    from tools.modules.unet.unet_unianimate import UNetSD_UniAnimate

    req_type_entry = eval(req_type)
    # req_type_entry = registry.get(req_type)

If you meet similar cases, try to import it explicitly. Hope this helps you.

@zhenyuanzhou you need try this.

zhenyuanzhou commented 4 days ago

Hi, you can try to change https://github.com/ali-vilab/UniAnimate/blob/549ee5fad7618500790929b0ae73151d36649045/utils/registry.py#L50-L53

to

req_type = cfg.pop("type") 
req_type_entry = req_type 
if isinstance(req_type, str): 
    from tools.inferences.inference_unianimate_entrance import inference_unianimate_entrance
    from tools.modules.diffusions.diffusion_ddim import DiffusionDDIM
    from tools.modules.autoencoder import AutoencoderKL
    from tools.modules.clip_embedder import FrozenOpenCLIPTextVisualEmbedder
    from tools.modules.unet.unet_unianimate import UNetSD_UniAnimate

    req_type_entry = eval(req_type)
    # req_type_entry = registry.get(req_type)

If you meet similar cases, try to import it explicitly. Hope this helps you.

@zhenyuanzhou you need try this.

All the problem solved. Thank you again for you patientce and kind help. Appreciate.

https://github.com/ali-vilab/UniAnimate/assets/7355407/ab253ade-62a6-426c-91a5-3744d592e80f