mmcm issues despite being installed properly (running through docker)

rmasiso commented 5 months ago

Hi! First I just want to say, thank you for making this opensource and doing such an amazing job with this.

I'm having no problems with text2video, but with video2video I'm having problems. Any of the video2video examples in the github readme don't work.

I'm running the example: python scripts/inference/video2video.py --sd_model_name fantasticmix_v10 --unet_model_name musev_referencenet --referencenet_model_name musev_referencenet --ip_adapter_model_name musev_referencenet -test_data_path ./configs/tasks/example.yaml --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder --output_dir ./output --n_batch 1 --controlnet_name dwpose_body_hand --which2video "video_middle" --target_datas dance1 --fps 12 --time_size 12

Heres' the full error:

(musev) root@10703020883e:/workspace/MuseV# python scripts/inference/video2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev_referencenet --referencenet_model_name   musev_referencenet --ip_adapter_model_name musev_referencenet    -test_data_path ./configs/tasks/example.yaml    --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder      --output_dir ./output  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video "video_middle"  --target_datas dance1 --fps 12 --time_size 12
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
2024-04-05 20:19:43.562604: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-05 20:19:43.584943: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-05 20:19:44.056965: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/workspace/MuseV/diffusers/src/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
args
{'add_static_video_prompt': False,
 'context_batch_size': 1,
 'context_frames': 12,
 'context_overlap': 4,
 'context_schedule': 'uniform_v2',
 'context_stride': 1,
 'controlnet_conditioning_scale': 1.0,
 'controlnet_name': 'dwpose_body_hand',
 'cross_attention_dim': 768,
 'enable_zero_snr': False,
 'end_to_end': True,
 'face_image_path': None,
 'facein_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/facein.py',
 'facein_model_name': None,
 'facein_scale': 1.0,
 'fix_condition_images': False,
 'fixed_ip_adapter_image': True,
 'fixed_refer_face_image': True,
 'fixed_refer_image': True,
 'fps': 12,
 'guidance_scale': 7.5,
 'height': None,
 'img_length_ratio': 1.0,
 'img_weight': 0.001,
 'interpolation_factor': 1,
 'ip_adapter_face_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_face_model_name': None,
 'ip_adapter_face_scale': 1.0,
 'ip_adapter_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_model_name': 'musev_referencenet',
 'ip_adapter_scale': 1.0,
 'ipadapter_image_path': None,
 'lcm_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/lcm_model.py',
 'lcm_model_name': None,
 'log_level': 'INFO',
 'motion_speed': 8.0,
 'n_batch': 1,
 'n_cols': 3,
 'n_repeat': 1,
 'n_vision_condition': 1,
 'need_hist_match': False,
 'need_img_based_video_noise': True,
 'need_return_condition': False,
 'need_return_videos': False,
 'need_video2video': False,
 'negative_prompt': 'V2',
 'negprompt_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/negative_prompt.py',
 'noise_type': 'video_fusion',
 'num_inference_steps': 30,
 'output_dir': './output',
 'overwrite': False,
 'pose_guider_model_path': None,
 'prompt_only_use_image_prompt': False,
 'record_mid_video_latents': False,
 'record_mid_video_noises': False,
 'redraw_condition_image': False,
 'redraw_condition_image_with_facein': True,
 'redraw_condition_image_with_ip_adapter_face': True,
 'redraw_condition_image_with_ipdapter': True,
 'redraw_condition_image_with_referencenet': True,
 'referencenet_image_path': None,
 'referencenet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/referencenet.py',
 'referencenet_model_name': 'musev_referencenet',
 'sample_rate': 1,
 'save_filetype': 'mp4',
 'save_images': False,
 'sd_model_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/T2I_all_model.py',
 'sd_model_name': 'fantasticmix_v10',
 'seed': None,
 'strength': 0.8,
 'target_datas': 'dance1',
 'test_data_path': './configs/tasks/example.yaml',
 'time_size': 12,
 'unet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/motion_model.py',
 'unet_model_name': 'musev_referencenet',
 'use_condition_image': True,
 'vae_model_path': './checkpoints/vae/sd-vae-ft-mse',
 'video_guidance_scale': 3.5,
 'video_guidance_scale_end': None,
 'video_guidance_scale_method': 'linear',
 'video_has_condition': True,
 'video_is_middle': False,
 'video_negative_prompt': 'V2',
 'video_num_inference_steps': 10,
 'video_overlap': 1,
 'video_strength': 1.0,
 'vision_clip_extractor_class_name': 'ImageClipVisionFeatureExtractor',
 'vision_clip_model_path': './checkpoints/IP-Adapter/models/image_encoder',
 'w_ind_noise': 0.5,
 'which2video': 'video_middle',
 'width': None,
 'write_info': False}

running model, T2I SD
{'fantasticmix_v10': {'sd': '/workspace/MuseV/configs/model/../../checkpoints/t2i/sd1.5/fantasticmix_v10'}}
lcm:  None None
unet_model_params_dict_src dict_keys(['musev', 'musev_referencenet', 'musev_referencenet_pose'])
unet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
referencenet_model_params_dict_src dict_keys(['musev_referencenet'])
referencenet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
ip_adapter_model_params_dict_src dict_keys(['IPAdapter', 'IPAdapterPlus', 'IPAdapterPlus-face', 'IPAdapterFaceID', 'musev_referencenet', 'musev_referencenet_pose'])
ip_adapter:  musev_referencenet {'ip_image_encoder': '/workspace/MuseV/configs/model/../../checkpoints/IP-Adapter/image_encoder', 'ip_ckpt': '/workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet/ip_adapter_image_proj.bin', 'ip_scale': 1.0, 'clip_extra_context_tokens': 4, 'clip_embeddings_dim': 1024, 'desp': ''}
facein:  None None
ip_adapter_face:  None None
video_negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
n_test_datas 1
2024-04-05 20:19:51,813- musev:997- INFO- vision_clip_extractor, name=ImageClipVisionFeatureExtractor, path=./checkpoints/IP-Adapter/models/image_encoder
test_model_vae_model_path ./checkpoints/vae/sd-vae-ft-mse
Traceback (most recent call last):
  File "/workspace/MuseV/scripts/inference/video2video.py", line 1102, in <module>
    sd_predictor = DiffusersPipelinePredictor(
  File "/workspace/MuseV/musev/pipelines/pipeline_controlnet_predictor.py", line 165, in __init__
    controlnet, controlnet_processor, processor_params = load_controlnet_model(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 856, in load_controlnet_model
    controlnet_processor = ControlnetProcessor(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 71, in __init__
    self.processor = processor_cls()
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 141, in __init__
    self.pose_estimation = Wholebody(
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py", line 53, in __init__
    self.detector = init_detector(det_config, det_ckpt, device=device)
NameError: name 'init_detector' is not defined

The error tells me there's no variable called "init_detector", so when we look into the code, it's defined here: /workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py

This line:

    from mmdet.apis import inference_detector, init_detector
except ImportError:
    warnings.warn(
        "The module 'mmdet' is not installed. The package will have limited functionality. Please install it using the command: mim install 'mmdet>=3.1.0'"
    )

Interestingly, I don't get a warning that "mmcv" is not installed, as you'd expect. But, the script believes that init_detector is not defined. So, I took out the line from mmdet.apis import inference_detector, init_detector out of the try statement to see what happened.

I still get an error, but a different one:

(musev) root@10703020883e:/workspace/MuseV# python scripts/inference/video2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev_referencenet --referencenet_model_name   musev_referencenet --ip_adapter_model_name musev_referencenet    -test_data_path ./configs/tasks/example.yaml    --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder      --output_dir ./output  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video "video_middle"  --target_datas dance1 --fps 12 --time_size 12
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
2024-04-05 20:26:05.951185: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-05 20:26:05.973544: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-05 20:26:06.415460: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/workspace/MuseV/diffusers/src/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
args
{'add_static_video_prompt': False,
 'context_batch_size': 1,
 'context_frames': 12,
 'context_overlap': 4,
 'context_schedule': 'uniform_v2',
 'context_stride': 1,
 'controlnet_conditioning_scale': 1.0,
 'controlnet_name': 'dwpose_body_hand',
 'cross_attention_dim': 768,
 'enable_zero_snr': False,
 'end_to_end': True,
 'face_image_path': None,
 'facein_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/facein.py',
 'facein_model_name': None,
 'facein_scale': 1.0,
 'fix_condition_images': False,
 'fixed_ip_adapter_image': True,
 'fixed_refer_face_image': True,
 'fixed_refer_image': True,
 'fps': 12,
 'guidance_scale': 7.5,
 'height': None,
 'img_length_ratio': 1.0,
 'img_weight': 0.001,
 'interpolation_factor': 1,
 'ip_adapter_face_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_face_model_name': None,
 'ip_adapter_face_scale': 1.0,
 'ip_adapter_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_model_name': 'musev_referencenet',
 'ip_adapter_scale': 1.0,
 'ipadapter_image_path': None,
 'lcm_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/lcm_model.py',
 'lcm_model_name': None,
 'log_level': 'INFO',
 'motion_speed': 8.0,
 'n_batch': 1,
 'n_cols': 3,
 'n_repeat': 1,
 'n_vision_condition': 1,
 'need_hist_match': False,
 'need_img_based_video_noise': True,
 'need_return_condition': False,
 'need_return_videos': False,
 'need_video2video': False,
 'negative_prompt': 'V2',
 'negprompt_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/negative_prompt.py',
 'noise_type': 'video_fusion',
 'num_inference_steps': 30,
 'output_dir': './output',
 'overwrite': False,
 'pose_guider_model_path': None,
 'prompt_only_use_image_prompt': False,
 'record_mid_video_latents': False,
 'record_mid_video_noises': False,
 'redraw_condition_image': False,
 'redraw_condition_image_with_facein': True,
 'redraw_condition_image_with_ip_adapter_face': True,
 'redraw_condition_image_with_ipdapter': True,
 'redraw_condition_image_with_referencenet': True,
 'referencenet_image_path': None,
 'referencenet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/referencenet.py',
 'referencenet_model_name': 'musev_referencenet',
 'sample_rate': 1,
 'save_filetype': 'mp4',
 'save_images': False,
 'sd_model_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/T2I_all_model.py',
 'sd_model_name': 'fantasticmix_v10',
 'seed': None,
 'strength': 0.8,
 'target_datas': 'dance1',
 'test_data_path': './configs/tasks/example.yaml',
 'time_size': 12,
 'unet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/motion_model.py',
 'unet_model_name': 'musev_referencenet',
 'use_condition_image': True,
 'vae_model_path': './checkpoints/vae/sd-vae-ft-mse',
 'video_guidance_scale': 3.5,
 'video_guidance_scale_end': None,
 'video_guidance_scale_method': 'linear',
 'video_has_condition': True,
 'video_is_middle': False,
 'video_negative_prompt': 'V2',
 'video_num_inference_steps': 10,
 'video_overlap': 1,
 'video_strength': 1.0,
 'vision_clip_extractor_class_name': 'ImageClipVisionFeatureExtractor',
 'vision_clip_model_path': './checkpoints/IP-Adapter/models/image_encoder',
 'w_ind_noise': 0.5,
 'which2video': 'video_middle',
 'width': None,
 'write_info': False}

running model, T2I SD
{'fantasticmix_v10': {'sd': '/workspace/MuseV/configs/model/../../checkpoints/t2i/sd1.5/fantasticmix_v10'}}
lcm:  None None
unet_model_params_dict_src dict_keys(['musev', 'musev_referencenet', 'musev_referencenet_pose'])
unet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
referencenet_model_params_dict_src dict_keys(['musev_referencenet'])
referencenet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
ip_adapter_model_params_dict_src dict_keys(['IPAdapter', 'IPAdapterPlus', 'IPAdapterPlus-face', 'IPAdapterFaceID', 'musev_referencenet', 'musev_referencenet_pose'])
ip_adapter:  musev_referencenet {'ip_image_encoder': '/workspace/MuseV/configs/model/../../checkpoints/IP-Adapter/image_encoder', 'ip_ckpt': '/workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet/ip_adapter_image_proj.bin', 'ip_scale': 1.0, 'clip_extra_context_tokens': 4, 'clip_embeddings_dim': 1024, 'desp': ''}
facein:  None None
ip_adapter_face:  None None
video_negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
n_test_datas 1
2024-04-05 20:26:13,677- musev:997- INFO- vision_clip_extractor, name=ImageClipVisionFeatureExtractor, path=./checkpoints/IP-Adapter/models/image_encoder
test_model_vae_model_path ./checkpoints/vae/sd-vae-ft-mse
Traceback (most recent call last):
  File "/workspace/MuseV/scripts/inference/video2video.py", line 1102, in <module>
    sd_predictor = DiffusersPipelinePredictor(
  File "/workspace/MuseV/musev/pipelines/pipeline_controlnet_predictor.py", line 165, in __init__
    controlnet, controlnet_processor, processor_params = load_controlnet_model(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 856, in load_controlnet_model
    controlnet_processor = ControlnetProcessor(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 71, in __init__
    self.processor = processor_cls()
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 139, in __init__
    from .wholebody import Wholebody
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py", line 24, in <module>
    from mmdet.apis import inference_detector, init_detector
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/apis/__init__.py", line 2, in <module>
    from .det_inferencer import DetInferencer
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/apis/det_inferencer.py", line 22, in <module>
    from mmdet.evaluation import INSTANCE_OFFSET
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/evaluation/__init__.py", line 3, in <module>
    from .metrics import *  # noqa: F401,F403
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/evaluation/metrics/__init__.py", line 5, in <module>
    from .coco_metric import CocoMetric
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/evaluation/metrics/coco_metric.py", line 16, in <module>
    from mmdet.datasets.api_wrappers import COCO, COCOeval, COCOevalMP
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/__init__.py", line 26, in <module>
    from .utils import get_loading_pipeline
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/utils.py", line 5, in <module>
    from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/transforms/__init__.py", line 6, in <module>
    from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs,
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/transforms/formatting.py", line 11, in <module>
    from mmdet.structures.bbox import BaseBoxes
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/bbox/__init__.py", line 2, in <module>
    from .base_boxes import BaseBoxes
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/bbox/base_boxes.py", line 9, in <module>
    from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/mask/__init__.py", line 3, in <module>
    from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks,
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/mask/structures.py", line 12, in <module>
    from mmcv.ops.roi_align import roi_align
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/__init__.py", line 3, in <module>
    from .active_rotated_filter import active_rotated_filter
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
    ext_module = ext_loader.load_ext(
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/opt/conda/envs/musev/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEENS6_INS5_12MemoryFormatEEE
(musev) root@10703020883e:/workspace/MuseV# python scripts/inference/video2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev_referencenet --referencenet_model_name   musev_referencenet --ip_adapter_model_name musev_referencenet    -test_data_path ./configs/tasks/example.yaml    --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder      --output_dir ./output  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video "video_middle"  --target_datas dance1 --fps 12 --time_size 12
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
2024-04-05 20:28:21.233723: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-05 20:28:21.261046: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-05 20:28:21.719853: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/workspace/MuseV/diffusers/src/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
args
{'add_static_video_prompt': False,
 'context_batch_size': 1,
 'context_frames': 12,
 'context_overlap': 4,
 'context_schedule': 'uniform_v2',
 'context_stride': 1,
 'controlnet_conditioning_scale': 1.0,
 'controlnet_name': 'dwpose_body_hand',
 'cross_attention_dim': 768,
 'enable_zero_snr': False,
 'end_to_end': True,
 'face_image_path': None,
 'facein_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/facein.py',
 'facein_model_name': None,
 'facein_scale': 1.0,
 'fix_condition_images': False,
 'fixed_ip_adapter_image': True,
 'fixed_refer_face_image': True,
 'fixed_refer_image': True,
 'fps': 12,
 'guidance_scale': 7.5,
 'height': None,
 'img_length_ratio': 1.0,
 'img_weight': 0.001,
 'interpolation_factor': 1,
 'ip_adapter_face_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_face_model_name': None,
 'ip_adapter_face_scale': 1.0,
 'ip_adapter_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_model_name': 'musev_referencenet',
 'ip_adapter_scale': 1.0,
 'ipadapter_image_path': None,
 'lcm_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/lcm_model.py',
 'lcm_model_name': None,
 'log_level': 'INFO',
 'motion_speed': 8.0,
 'n_batch': 1,
 'n_cols': 3,
 'n_repeat': 1,
 'n_vision_condition': 1,
 'need_hist_match': False,
 'need_img_based_video_noise': True,
 'need_return_condition': False,
 'need_return_videos': False,
 'need_video2video': False,
 'negative_prompt': 'V2',
 'negprompt_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/negative_prompt.py',
 'noise_type': 'video_fusion',
 'num_inference_steps': 30,
 'output_dir': './output',
 'overwrite': False,
 'pose_guider_model_path': None,
 'prompt_only_use_image_prompt': False,
 'record_mid_video_latents': False,
 'record_mid_video_noises': False,
 'redraw_condition_image': False,
 'redraw_condition_image_with_facein': True,
 'redraw_condition_image_with_ip_adapter_face': True,
 'redraw_condition_image_with_ipdapter': True,
 'redraw_condition_image_with_referencenet': True,
 'referencenet_image_path': None,
 'referencenet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/referencenet.py',
 'referencenet_model_name': 'musev_referencenet',
 'sample_rate': 1,
 'save_filetype': 'mp4',
 'save_images': False,
 'sd_model_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/T2I_all_model.py',
 'sd_model_name': 'fantasticmix_v10',
 'seed': None,
 'strength': 0.8,
 'target_datas': 'dance1',
 'test_data_path': './configs/tasks/example.yaml',
 'time_size': 12,
 'unet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/motion_model.py',
 'unet_model_name': 'musev_referencenet',
 'use_condition_image': True,
 'vae_model_path': './checkpoints/vae/sd-vae-ft-mse',
 'video_guidance_scale': 3.5,
 'video_guidance_scale_end': None,
 'video_guidance_scale_method': 'linear',
 'video_has_condition': True,
 'video_is_middle': False,
 'video_negative_prompt': 'V2',
 'video_num_inference_steps': 10,
 'video_overlap': 1,
 'video_strength': 1.0,
 'vision_clip_extractor_class_name': 'ImageClipVisionFeatureExtractor',
 'vision_clip_model_path': './checkpoints/IP-Adapter/models/image_encoder',
 'w_ind_noise': 0.5,
 'which2video': 'video_middle',
 'width': None,
 'write_info': False}

running model, T2I SD
{'fantasticmix_v10': {'sd': '/workspace/MuseV/configs/model/../../checkpoints/t2i/sd1.5/fantasticmix_v10'}}
lcm:  None None
unet_model_params_dict_src dict_keys(['musev', 'musev_referencenet', 'musev_referencenet_pose'])
unet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
referencenet_model_params_dict_src dict_keys(['musev_referencenet'])
referencenet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
ip_adapter_model_params_dict_src dict_keys(['IPAdapter', 'IPAdapterPlus', 'IPAdapterPlus-face', 'IPAdapterFaceID', 'musev_referencenet', 'musev_referencenet_pose'])
ip_adapter:  musev_referencenet {'ip_image_encoder': '/workspace/MuseV/configs/model/../../checkpoints/IP-Adapter/image_encoder', 'ip_ckpt': '/workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet/ip_adapter_image_proj.bin', 'ip_scale': 1.0, 'clip_extra_context_tokens': 4, 'clip_embeddings_dim': 1024, 'desp': ''}
facein:  None None
ip_adapter_face:  None None
video_negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
n_test_datas 1
2024-04-05 20:28:28,992- musev:997- INFO- vision_clip_extractor, name=ImageClipVisionFeatureExtractor, path=./checkpoints/IP-Adapter/models/image_encoder
test_model_vae_model_path ./checkpoints/vae/sd-vae-ft-mse
Traceback (most recent call last):
  File "/workspace/MuseV/scripts/inference/video2video.py", line 1102, in <module>
    sd_predictor = DiffusersPipelinePredictor(
  File "/workspace/MuseV/musev/pipelines/pipeline_controlnet_predictor.py", line 165, in __init__
    controlnet, controlnet_processor, processor_params = load_controlnet_model(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 856, in load_controlnet_model
    controlnet_processor = ControlnetProcessor(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 71, in __init__
    self.processor = processor_cls()
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 139, in __init__
    from .wholebody import Wholebody
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py", line 13, in <module>
    from mmpose.apis import inference_topdown
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/apis/__init__.py", line 2, in <module>
    from .inference import (collect_multi_frames, inference_bottomup,
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/apis/inference.py", line 17, in <module>
    from mmpose.models.builder import build_pose_estimator
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/__init__.py", line 8, in <module>
    from .heads import *  # noqa
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/heads/__init__.py", line 11, in <module>
    from .transformer_heads import EDPoseHead
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/heads/transformer_heads/__init__.py", line 2, in <module>
    from .edpose_head import EDPoseHead
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/heads/transformer_heads/edpose_head.py", line 14, in <module>
    from mmcv.ops import MultiScaleDeformableAttention
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/__init__.py", line 3, in <module>
    from .active_rotated_filter import active_rotated_filter
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
    ext_module = ext_loader.load_ext(
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/opt/conda/envs/musev/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEENS6_INS5_12MemoryFormatEEE
(musev) root@10703020883e:/workspace/MuseV# python scripts/inference/video2video.py --sd_model_name fantasticmix_v10  --unet_model_name musev_referencenet --referencenet_model_name   musev_referencenet --ip_adapter_model_name musev_referencenet    -test_data_path ./configs/tasks/example.yaml    --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder      --output_dir ./output  --n_batch 1 --controlnet_name dwpose_body_hand  --which2video "video_middle"  --target_datas dance1 --fps 12 --time_size 12
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
2024-04-05 20:40:39.017376: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-05 20:40:39.041670: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-05 20:40:39.480456: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/opt/conda/envs/musev/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
/workspace/MuseV/diffusers/src/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  torch.utils._pytree._register_pytree_node(
args
{'add_static_video_prompt': False,
 'context_batch_size': 1,
 'context_frames': 12,
 'context_overlap': 4,
 'context_schedule': 'uniform_v2',
 'context_stride': 1,
 'controlnet_conditioning_scale': 1.0,
 'controlnet_name': 'dwpose_body_hand',
 'cross_attention_dim': 768,
 'enable_zero_snr': False,
 'end_to_end': True,
 'face_image_path': None,
 'facein_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/facein.py',
 'facein_model_name': None,
 'facein_scale': 1.0,
 'fix_condition_images': False,
 'fixed_ip_adapter_image': True,
 'fixed_refer_face_image': True,
 'fixed_refer_image': True,
 'fps': 12,
 'guidance_scale': 7.5,
 'height': None,
 'img_length_ratio': 1.0,
 'img_weight': 0.001,
 'interpolation_factor': 1,
 'ip_adapter_face_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_face_model_name': None,
 'ip_adapter_face_scale': 1.0,
 'ip_adapter_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/ip_adapter.py',
 'ip_adapter_model_name': 'musev_referencenet',
 'ip_adapter_scale': 1.0,
 'ipadapter_image_path': None,
 'lcm_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/lcm_model.py',
 'lcm_model_name': None,
 'log_level': 'INFO',
 'motion_speed': 8.0,
 'n_batch': 1,
 'n_cols': 3,
 'n_repeat': 1,
 'n_vision_condition': 1,
 'need_hist_match': False,
 'need_img_based_video_noise': True,
 'need_return_condition': False,
 'need_return_videos': False,
 'need_video2video': False,
 'negative_prompt': 'V2',
 'negprompt_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/negative_prompt.py',
 'noise_type': 'video_fusion',
 'num_inference_steps': 30,
 'output_dir': './output',
 'overwrite': False,
 'pose_guider_model_path': None,
 'prompt_only_use_image_prompt': False,
 'record_mid_video_latents': False,
 'record_mid_video_noises': False,
 'redraw_condition_image': False,
 'redraw_condition_image_with_facein': True,
 'redraw_condition_image_with_ip_adapter_face': True,
 'redraw_condition_image_with_ipdapter': True,
 'redraw_condition_image_with_referencenet': True,
 'referencenet_image_path': None,
 'referencenet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/referencenet.py',
 'referencenet_model_name': 'musev_referencenet',
 'sample_rate': 1,
 'save_filetype': 'mp4',
 'save_images': False,
 'sd_model_cfg_path': '/workspace/MuseV/scripts/inference/../../configs/model/T2I_all_model.py',
 'sd_model_name': 'fantasticmix_v10',
 'seed': None,
 'strength': 0.8,
 'target_datas': 'dance1',
 'test_data_path': './configs/tasks/example.yaml',
 'time_size': 12,
 'unet_model_cfg_path': '/workspace/MuseV/scripts/inference/../.././configs/model/motion_model.py',
 'unet_model_name': 'musev_referencenet',
 'use_condition_image': True,
 'vae_model_path': './checkpoints/vae/sd-vae-ft-mse',
 'video_guidance_scale': 3.5,
 'video_guidance_scale_end': None,
 'video_guidance_scale_method': 'linear',
 'video_has_condition': True,
 'video_is_middle': False,
 'video_negative_prompt': 'V2',
 'video_num_inference_steps': 10,
 'video_overlap': 1,
 'video_strength': 1.0,
 'vision_clip_extractor_class_name': 'ImageClipVisionFeatureExtractor',
 'vision_clip_model_path': './checkpoints/IP-Adapter/models/image_encoder',
 'w_ind_noise': 0.5,
 'which2video': 'video_middle',
 'width': None,
 'write_info': False}

running model, T2I SD
{'fantasticmix_v10': {'sd': '/workspace/MuseV/configs/model/../../checkpoints/t2i/sd1.5/fantasticmix_v10'}}
lcm:  None None
unet_model_params_dict_src dict_keys(['musev', 'musev_referencenet', 'musev_referencenet_pose'])
unet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
referencenet_model_params_dict_src dict_keys(['musev_referencenet'])
referencenet:  musev_referencenet /workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet
ip_adapter_model_params_dict_src dict_keys(['IPAdapter', 'IPAdapterPlus', 'IPAdapterPlus-face', 'IPAdapterFaceID', 'musev_referencenet', 'musev_referencenet_pose'])
ip_adapter:  musev_referencenet {'ip_image_encoder': '/workspace/MuseV/configs/model/../../checkpoints/IP-Adapter/image_encoder', 'ip_ckpt': '/workspace/MuseV/configs/model/../../checkpoints/motion/musev_referencenet/ip_adapter_image_proj.bin', 'ip_scale': 1.0, 'clip_extra_context_tokens': 4, 'clip_embeddings_dim': 1024, 'desp': ''}
facein:  None None
ip_adapter_face:  None None
video_negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
negprompt V2 badhandv4, ng_deepnegative_v1_75t, (((multiple heads))), (((bad body))), (((two people))), ((extra arms)), ((deformed body)), (((sexy))), paintings,(((two heads))), ((big head)),sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, glans, (((nsfw))), nipples, extra fingers, (extra legs), (long neck), mutated hands, (fused fingers), (too many fingers)
n_test_datas 1
2024-04-05 20:40:46,818- musev:997- INFO- vision_clip_extractor, name=ImageClipVisionFeatureExtractor, path=./checkpoints/IP-Adapter/models/image_encoder
test_model_vae_model_path ./checkpoints/vae/sd-vae-ft-mse
Traceback (most recent call last):
  File "/workspace/MuseV/scripts/inference/video2video.py", line 1102, in <module>
    sd_predictor = DiffusersPipelinePredictor(
  File "/workspace/MuseV/musev/pipelines/pipeline_controlnet_predictor.py", line 165, in __init__
    controlnet, controlnet_processor, processor_params = load_controlnet_model(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 856, in load_controlnet_model
    controlnet_processor = ControlnetProcessor(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 71, in __init__
    self.processor = processor_cls()
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 139, in __init__
    from .wholebody import Wholebody
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py", line 25, in <module>
    from mmdet.apis import inference_detector, init_detector
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/apis/__init__.py", line 2, in <module>
    from .det_inferencer import DetInferencer
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/apis/det_inferencer.py", line 22, in <module>
    from mmdet.evaluation import INSTANCE_OFFSET
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/evaluation/__init__.py", line 3, in <module>
    from .metrics import *  # noqa: F401,F403
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/evaluation/metrics/__init__.py", line 5, in <module>
    from .coco_metric import CocoMetric
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/evaluation/metrics/coco_metric.py", line 16, in <module>
    from mmdet.datasets.api_wrappers import COCO, COCOeval, COCOevalMP
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/__init__.py", line 26, in <module>
    from .utils import get_loading_pipeline
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/utils.py", line 5, in <module>
    from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/transforms/__init__.py", line 6, in <module>
    from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs,
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/datasets/transforms/formatting.py", line 11, in <module>
    from mmdet.structures.bbox import BaseBoxes
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/bbox/__init__.py", line 2, in <module>
    from .base_boxes import BaseBoxes
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/bbox/base_boxes.py", line 9, in <module>
    from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/mask/__init__.py", line 3, in <module>
    from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks,
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmdet/structures/mask/structures.py", line 12, in <module>
    from mmcv.ops.roi_align import roi_align
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/__init__.py", line 3, in <module>
    from .active_rotated_filter import active_rotated_filter
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
    ext_module = ext_loader.load_ext(
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/opt/conda/envs/musev/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEENS6_INS5_12MemoryFormatEEE

Any help would be appreciated!

itechmusic commented 5 months ago

@rmasiso hi，you can try install mediapipe

xzqjack commented 5 months ago

@rmasiso hi, init_detector is in openmim package, which should be all right in docker. However, you can try install these packages manually

pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0"

rmasiso commented 5 months ago

Thanks for responding. I had tested the above solutions previously, but I did it again and they all resulted in: "requirement satisfied". So, it didn't change anything.

I will add that I conda installed torch to work with cuda12.1 -- but, I don't think this is a problem because the text2image example works and I can use other controlnets.

If I change the controlnet from dwpose_hand_body to pose_hand_body or something that is not dwpose associated, it seems to work. I haven't tested all the controlnets though (tested canny, deptch, pose_hand_body, pose_body -- and they work). Why isn't dwpose working then?

In other words, this works: python scripts/inference/video2video.py --sd_model_name fantasticmix_v10 --unet_model_name musev_referencenet --referencenet_model_name musev_referencenet --ip_adapter_model_name musev_referencenet -test_data_path ./configs/tasks/example.yaml --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder --output_dir ./output --n_batch 1 --controlnet_name pose_hand_body --which2video "video_middle" --target_datas dance1 --fps 12 --time_size 48

this does not work: python scripts/inference/video2video.py --sd_model_name fantasticmix_v10 --unet_model_name musev_referencenet --referencenet_model_name musev_referencenet --ip_adapter_model_name musev_referencenet -test_data_path ./configs/tasks/example.yaml --vision_clip_extractor_class_name ImageClipVisionFeatureExtractor --vision_clip_model_path ./checkpoints/IP-Adapter/models/image_encoder --output_dir ./output --n_batch 1 --controlnet_name dwpose_hand_body --which2video "video_middle" --target_datas dance1 --fps 12 --time_size 48

Thanks again!

xzqjack commented 5 months ago

@rmasiso dwpose rely on mmpose, openpose define model in controlnet_aux and doesn't rely on other packages.

Can you check the conda env path?

conda env list 
pip list |grep mmpose

For some reasons we are not aware of, conda python cannot find openmin. You can try reinstall these with bellow code, and check whether these package are installed in conda musev env.

pip install --no-cache-dir -U openmim 
mim install -U mmengine 
mim install  -U "mmcv>=2.0.1" 
mim install  -U "mmdet>=3.1.0" 
mim install  -U "mmpose>=1.1.0"

rmasiso commented 5 months ago

Here's the conda env list

(musev) root@2f82e80bc1d7:/workspace/MuseV# conda env list
mmpose# conda environments:
#
base                     /opt/conda
musev                 *  /opt/conda/envs/musev

Here's the grep:

(musev) root@2f82e80bc1d7:/workspace/MuseV# pip list |grep mmpose
mmpose                       1.3.1

and then here's the mim installs:

(musev) root@2f82e80bc1d7:/workspace/MuseV# pip install --no-cache-dir -U openmim
mim install -U mmengine
mim install  -U "mmcv>=2.0.1"
mim install  -U "mmdet>=3.1.0"
mim install  -U "mmpose>=1.1.0" Requirement already satisfied: openmim in /opt/conda/envs/musev/lib/python3.10/site-packages (0.3.9)
Requirement already satisfied: Click in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (8.1.7)
Requirement already satisfied: colorama in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (0.4.6)
Requirement already satisfied: model-index in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (0.1.11)
Requirement already satisfied: opendatalab in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (0.0.10)
Requirement already satisfied: pandas in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (2.2.1)
Requirement already satisfied: pip>=19.3 in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (23.3.1)
Requirement already satisfied: requests in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (2.28.2)
Requirement already satisfied: rich in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (13.4.2)
Requirement already satisfied: tabulate in /opt/conda/envs/musev/lib/python3.10/site-packages (from openmim) (0.9.0)
Requirement already satisfied: pyyaml in /opt/conda/envs/musev/lib/python3.10/site-packages (from model-index->openmim) (6.0.1)
Requirement already satisfied: markdown in /opt/conda/envs/musev/lib/python3.10/site-packages (from model-index->openmim) (3.6)
Requirement already satisfied: ordered-set in /opt/conda/envs/musev/lib/python3.10/site-packages (from model-index->openmim) (4.1.0)
Requirement already satisfied: pycryptodome in /opt/conda/envs/musev/lib/python3.10/site-packages (from opendatalab->openmim) (3.20.0)
Requirement already satisfied: tqdm in /opt/conda/envs/musev/lib/python3.10/site-packages (from opendatalab->openmim) (4.65.2)
Requirement already satisfied: openxlab in /opt/conda/envs/musev/lib/python3.10/site-packages (from opendatalab->openmim) (0.0.36)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->openmim) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->openmim) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->openmim) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->openmim) (2024.2.2)
Requirement already satisfied: numpy<2,>=1.22.4 in /opt/conda/envs/musev/lib/python3.10/site-packages (from pandas->openmim) (1.24.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/musev/lib/python3.10/site-packages (from pandas->openmim) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from pandas->openmim) (2023.4)
Requirement already satisfied: tzdata>=2022.7 in /opt/conda/envs/musev/lib/python3.10/site-packages (from pandas->openmim) (2024.1)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->openmim) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->openmim) (2.17.2)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->openmim) (0.1.2)
Requirement already satisfied: six>=1.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->openmim) (1.16.0)
Requirement already satisfied: oss2~=2.17.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from openxlab->opendatalab->openmim) (2.17.0)
Requirement already satisfied: setuptools~=60.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from openxlab->opendatalab->openmim) (60.2.0)
Requirement already satisfied: crcmod>=1.7 in /opt/conda/envs/musev/lib/python3.10/site-packages (from oss2~=2.17.0->openxlab->opendatalab->openmim) (1.7)
Requirement already satisfied: aliyun-python-sdk-kms>=2.4.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from oss2~=2.17.0->openxlab->opendatalab->openmim) (2.16.2)
Requirement already satisfied: aliyun-python-sdk-core>=2.13.12 in /opt/conda/envs/musev/lib/python3.10/site-packages (from oss2~=2.17.0->openxlab->opendatalab->openmim) (2.15.0)
Requirement already satisfied: jmespath<1.0.0,>=0.9.3 in /opt/conda/envs/musev/lib/python3.10/site-packages (from aliyun-python-sdk-core>=2.13.12->oss2~=2.17.0->openxlab->opendatalab->openmim) (0.10.0)
Requirement already satisfied: cryptography>=2.6.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from aliyun-python-sdk-core>=2.13.12->oss2~=2.17.0->openxlab->opendatalab->openmim) (42.0.5)
Requirement already satisfied: cffi>=1.12 in /opt/conda/envs/musev/lib/python3.10/site-packages (from cryptography>=2.6.0->aliyun-python-sdk-core>=2.13.12->oss2~=2.17.0->openxlab->opendatalab->openmim) (1.16.0)
Requirement already satisfied: pycparser in /opt/conda/envs/musev/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.6.0->aliyun-python-sdk-core>=2.13.12->oss2~=2.17.0->openxlab->opendatalab->openmim) (2.21)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(musev) root@2f82e80bc1d7:/workspace/MuseV# mim install -U mmengine
Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.2.0/index.html
Requirement already satisfied: mmengine in /opt/conda/envs/musev/lib/python3.10/site-packages (0.10.3)
Requirement already satisfied: addict in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (2.4.0)
Requirement already satisfied: matplotlib in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (3.6.2)
Requirement already satisfied: numpy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (1.24.3)
Requirement already satisfied: pyyaml in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (6.0.1)
Requirement already satisfied: rich in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (13.4.2)
Requirement already satisfied: termcolor in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (2.4.0)
Requirement already satisfied: yapf in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (0.40.2)
Requirement already satisfied: opencv-python>=3 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine) (4.9.0.80)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (4.50.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (1.4.5)
Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (10.2.0)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine) (2.9.0.post0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine) (2.17.2)
Requirement already satisfied: importlib-metadata>=6.6.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmengine) (7.1.0)
Requirement already satisfied: platformdirs>=3.5.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmengine) (4.2.0)
Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmengine) (2.0.1)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from importlib-metadata>=6.6.0->yapf->mmengine) (3.18.1)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->mmengine) (0.1.2)
Requirement already satisfied: six>=1.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->mmengine) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(musev) root@2f82e80bc1d7:/workspace/MuseV# mim install  -U "mmcv>=2.0.1"
Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.2.0/index.html
Requirement already satisfied: mmcv>=2.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (2.1.0)
Requirement already satisfied: addict in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (2.4.0)
Requirement already satisfied: mmengine>=0.3.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (0.10.3)
Requirement already satisfied: numpy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (1.24.3)
Requirement already satisfied: packaging in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (23.2)
Requirement already satisfied: Pillow in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (10.2.0)
Requirement already satisfied: pyyaml in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (6.0.1)
Requirement already satisfied: yapf in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (0.40.2)
Requirement already satisfied: opencv-python>=3 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv>=2.0.1) (4.9.0.80)
Requirement already satisfied: matplotlib in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine>=0.3.0->mmcv>=2.0.1) (3.6.2)
Requirement already satisfied: rich in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine>=0.3.0->mmcv>=2.0.1) (13.4.2)
Requirement already satisfied: termcolor in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine>=0.3.0->mmcv>=2.0.1) (2.4.0)
Requirement already satisfied: importlib-metadata>=6.6.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv>=2.0.1) (7.1.0)
Requirement already satisfied: platformdirs>=3.5.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv>=2.0.1) (4.2.0)
Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv>=2.0.1) (2.0.1)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from importlib-metadata>=6.6.0->yapf->mmcv>=2.0.1) (3.18.1)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (4.50.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (1.4.5)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (2.9.0.post0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine>=0.3.0->mmcv>=2.0.1) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine>=0.3.0->mmcv>=2.0.1) (2.17.2)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->mmengine>=0.3.0->mmcv>=2.0.1) (0.1.2)
Requirement already satisfied: six>=1.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->mmengine>=0.3.0->mmcv>=2.0.1) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(musev) root@2f82e80bc1d7:/workspace/MuseV# mim install  -U "mmdet>=3.1.0"
Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.2.0/index.html
Requirement already satisfied: mmdet>=3.1.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (3.2.0)
Collecting mmdet>=3.1.0
  Using cached mmdet-3.3.0-py3-none-any.whl.metadata (29 kB)
Requirement already satisfied: matplotlib in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (3.6.2)
Requirement already satisfied: numpy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (1.24.3)
Requirement already satisfied: pycocotools in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (2.0.7)
Requirement already satisfied: scipy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (1.12.0)
Requirement already satisfied: shapely in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (2.0.3)
Requirement already satisfied: six in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (1.16.0)
Requirement already satisfied: terminaltables in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (3.1.10)
Requirement already satisfied: tqdm in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (4.65.2)
Requirement already satisfied: mmcv<2.2.0,>=2.0.0rc4 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (2.1.0)
Requirement already satisfied: mmengine<1.0.0,>=0.7.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet>=3.1.0) (0.10.3)
Requirement already satisfied: addict in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (2.4.0)
Requirement already satisfied: packaging in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (23.2)
Requirement already satisfied: Pillow in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (10.2.0)
Requirement already satisfied: pyyaml in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (6.0.1)
Requirement already satisfied: yapf in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (0.40.2)
Requirement already satisfied: opencv-python>=3 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (4.9.0.80)
Requirement already satisfied: rich in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine<1.0.0,>=0.7.1->mmdet>=3.1.0) (13.4.2)
Requirement already satisfied: termcolor in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine<1.0.0,>=0.7.1->mmdet>=3.1.0) (2.4.0)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmdet>=3.1.0) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmdet>=3.1.0) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmdet>=3.1.0) (4.50.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmdet>=3.1.0) (1.4.5)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmdet>=3.1.0) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmdet>=3.1.0) (2.9.0.post0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine<1.0.0,>=0.7.1->mmdet>=3.1.0) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine<1.0.0,>=0.7.1->mmdet>=3.1.0) (2.17.2)
Requirement already satisfied: importlib-metadata>=6.6.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (7.1.0)
Requirement already satisfied: platformdirs>=3.5.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (4.2.0)
Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (2.0.1)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from importlib-metadata>=6.6.0->yapf->mmcv<2.2.0,>=2.0.0rc4->mmdet>=3.1.0) (3.18.1)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->mmengine<1.0.0,>=0.7.1->mmdet>=3.1.0) (0.1.2)
Using cached mmdet-3.3.0-py3-none-any.whl (2.2 MB)
Installing collected packages: mmdet
  Attempting uninstall: mmdet
    Found existing installation: mmdet 3.2.0
    Uninstalling mmdet-3.2.0:
      Successfully uninstalled mmdet-3.2.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mmpose 1.3.1 requires mmdet<3.3.0,>=3.0.0, but you have mmdet 3.3.0 which is incompatible.
Successfully installed mmdet-3.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(musev) root@2f82e80bc1d7:/workspace/MuseV# mim install  -U "mmpose>=1.1.0"
Looking in links: https://download.openmmlab.com/mmcv/dist/cu121/torch2.2.0/index.html
Requirement already satisfied: mmpose>=1.1.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (1.3.1)
Requirement already satisfied: chumpy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (0.70)
Requirement already satisfied: json-tricks in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (3.17.3)
Requirement already satisfied: matplotlib in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (3.6.2)
Requirement already satisfied: munkres in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (1.1.4)
Requirement already satisfied: numpy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (1.24.3)
Requirement already satisfied: opencv-python in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (4.9.0.80)
Requirement already satisfied: pillow in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (10.2.0)
Requirement already satisfied: scipy in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (1.12.0)
Requirement already satisfied: torchvision in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (0.15.2+cu118)
Requirement already satisfied: xtcocotools>=1.12 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (1.14.3)
Requirement already satisfied: mmcv<2.2.0,>=2.0.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (2.1.0)
Collecting mmdet<3.3.0,>=3.0.0 (from mmpose>=1.1.0)
  Using cached mmdet-3.2.0-py3-none-any.whl.metadata (32 kB)
Requirement already satisfied: mmengine<1.0.0,>=0.4.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmpose>=1.1.0) (0.10.3)
Requirement already satisfied: addict in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (2.4.0)
Requirement already satisfied: packaging in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (23.2)
Requirement already satisfied: pyyaml in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (6.0.1)
Requirement already satisfied: yapf in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (0.40.2)
Requirement already satisfied: pycocotools in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet<3.3.0,>=3.0.0->mmpose>=1.1.0) (2.0.7)
Requirement already satisfied: shapely in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet<3.3.0,>=3.0.0->mmpose>=1.1.0) (2.0.3)
Requirement already satisfied: six in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet<3.3.0,>=3.0.0->mmpose>=1.1.0) (1.16.0)
Requirement already satisfied: terminaltables in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet<3.3.0,>=3.0.0->mmpose>=1.1.0) (3.1.10)
Requirement already satisfied: tqdm in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmdet<3.3.0,>=3.0.0->mmpose>=1.1.0) (4.65.2)
Requirement already satisfied: rich in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine<1.0.0,>=0.4.0->mmpose>=1.1.0) (13.4.2)
Requirement already satisfied: termcolor in /opt/conda/envs/musev/lib/python3.10/site-packages (from mmengine<1.0.0,>=0.4.0->mmpose>=1.1.0) (2.4.0)
Requirement already satisfied: setuptools>=18.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from xtcocotools>=1.12->mmpose>=1.1.0) (60.2.0)
Requirement already satisfied: cython>=0.27.3 in /opt/conda/envs/musev/lib/python3.10/site-packages (from xtcocotools>=1.12->mmpose>=1.1.0) (3.0.2)
Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmpose>=1.1.0) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmpose>=1.1.0) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmpose>=1.1.0) (4.50.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmpose>=1.1.0) (1.4.5)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmpose>=1.1.0) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/envs/musev/lib/python3.10/site-packages (from matplotlib->mmpose>=1.1.0) (2.9.0.post0)
Requirement already satisfied: requests in /opt/conda/envs/musev/lib/python3.10/site-packages (from torchvision->mmpose>=1.1.0) (2.28.2)
Requirement already satisfied: torch==2.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from torchvision->mmpose>=1.1.0) (2.0.1+cu118)
Requirement already satisfied: filelock in /opt/conda/envs/musev/lib/python3.10/site-packages (from torch==2.0.1->torchvision->mmpose>=1.1.0) (3.13.1)
Requirement already satisfied: typing-extensions in /opt/conda/envs/musev/lib/python3.10/site-packages (from torch==2.0.1->torchvision->mmpose>=1.1.0) (4.10.0)
Requirement already satisfied: sympy in /opt/conda/envs/musev/lib/python3.10/site-packages (from torch==2.0.1->torchvision->mmpose>=1.1.0) (1.12)
Requirement already satisfied: networkx in /opt/conda/envs/musev/lib/python3.10/site-packages (from torch==2.0.1->torchvision->mmpose>=1.1.0) (3.1)
Requirement already satisfied: jinja2 in /opt/conda/envs/musev/lib/python3.10/site-packages (from torch==2.0.1->torchvision->mmpose>=1.1.0) (3.1.2)
Requirement already satisfied: triton==2.0.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from torch==2.0.1->torchvision->mmpose>=1.1.0) (2.0.0)
Requirement already satisfied: cmake in /opt/conda/envs/musev/lib/python3.10/site-packages (from triton==2.0.0->torch==2.0.1->torchvision->mmpose>=1.1.0) (3.25.0)
Requirement already satisfied: lit in /opt/conda/envs/musev/lib/python3.10/site-packages (from triton==2.0.0->torch==2.0.1->torchvision->mmpose>=1.1.0) (15.0.7)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->torchvision->mmpose>=1.1.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->torchvision->mmpose>=1.1.0) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->torchvision->mmpose>=1.1.0) (1.26.18)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/musev/lib/python3.10/site-packages (from requests->torchvision->mmpose>=1.1.0) (2024.2.2)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine<1.0.0,>=0.4.0->mmpose>=1.1.0) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from rich->mmengine<1.0.0,>=0.4.0->mmpose>=1.1.0) (2.17.2)
Requirement already satisfied: importlib-metadata>=6.6.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (7.1.0)
Requirement already satisfied: platformdirs>=3.5.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (4.2.0)
Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from yapf->mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (2.0.1)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/musev/lib/python3.10/site-packages (from importlib-metadata>=6.6.0->yapf->mmcv<2.2.0,>=2.0.0->mmpose>=1.1.0) (3.18.1)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/envs/musev/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->mmengine<1.0.0,>=0.4.0->mmpose>=1.1.0) (0.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/musev/lib/python3.10/site-packages (from jinja2->torch==2.0.1->torchvision->mmpose>=1.1.0) (2.0.1)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/musev/lib/python3.10/site-packages (from sympy->torch==2.0.1->torchvision->mmpose>=1.1.0) (1.3.0)
Using cached mmdet-3.2.0-py3-none-any.whl (2.1 MB)
Installing collected packages: mmdet
  Attempting uninstall: mmdet
    Found existing installation: mmdet 3.3.0
    Uninstalling mmdet-3.3.0:
      Successfully uninstalled mmdet-3.3.0
Successfully installed mmdet-3.2.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(musev) root@2f82e80bc1d7:/workspace/MuseV#

It uninstalls mmdet-3.2.0 and updates to mmdet-3.3.0 but then it uninstalls it again and installs 3.2 after running mmpose>=1.1.0

i still get the same error when running the dwpose:

Traceback (most recent call last):
  File "/workspace/MuseV/scripts/inference/video2video.py", line 1102, in <module>
    sd_predictor = DiffusersPipelinePredictor(
  File "/workspace/MuseV/musev/pipelines/pipeline_controlnet_predictor.py", line 165, in __init__
    controlnet, controlnet_processor, processor_params = load_controlnet_model(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 856, in load_controlnet_model
    controlnet_processor = ControlnetProcessor(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 71, in __init__
    self.processor = processor_cls()
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 141, in __init__
    self.pose_estimation = Wholebody(
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py", line 53, in __init__
    self.detector = init_detector(det_config, det_ckpt, device=device)
NameError: name 'init_detector' is not defined

Should I fix to a particular installation?

itechmusic commented 5 months ago

It seems the openmim is all right.

What's your pc system? Have you re-pull image, or re-run docker image？

itechmusic commented 5 months ago

@rmasiso To figure out, you can try look into whole_body.py, and move try, leave from import only, and then see detail error.

rmasiso commented 5 months ago

I did run a clean install of the docker image and ran everything again but ended up with the same issues.

I also did try to remove the try statements before and that changed the error, which I can report here.

This is what I modified in /workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py:

import os
import numpy as np
import warnings

import mmcv
# try:
#     import mmcv
# except ImportError:
#     warnings.warn(
#         "The module 'mmcv' is not installed. The package will have limited functionality. Please install it using the command: mim install 'mmcv>=2.0.1'"
#     )

from mmpose.apis import inference_topdown
from mmpose.apis import init_model as init_pose_estimator
from mmpose.evaluation.functional import nms
from mmpose.utils import adapt_mmdet_pipeline
from mmpose.structures import merge_data_samples

# try:
#     from mmpose.apis import inference_topdown
#     from mmpose.apis import init_model as init_pose_estimator
#     from mmpose.evaluation.functional import nms
#     from mmpose.utils import adapt_mmdet_pipeline
#     from mmpose.structures import merge_data_samples
# except ImportError:
#     warnings.warn(
#         "The module 'mmpose' is not installed. The package will have limited functionality. Please install it using the command: mim install 'mmpose>=1.1.0'"
#     )

from mmdet.apis import inference_detector, init_detector

# try:
#     from mmdet.apis import inference_detector, init_detector
# except ImportError:
#     warnings.warn(
#         "The module 'mmdet' is not installed. The package will have limited functionality. Please install it using the command: mim install 'mmdet>=3.1.0'"
#     )

This is the error after trying the first video2video example:

Traceback (most recent call last):
  File "/workspace/MuseV/scripts/inference/video2video.py", line 1102, in <module>
    sd_predictor = DiffusersPipelinePredictor(
  File "/workspace/MuseV/musev/pipelines/pipeline_controlnet_predictor.py", line 165, in __init__
    controlnet, controlnet_processor, processor_params = load_controlnet_model(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 856, in load_controlnet_model
    controlnet_processor = ControlnetProcessor(
  File "/workspace/MuseV/MMCM/mmcm/vision/feature_extractor/controlnet.py", line 71, in __init__
    self.processor = processor_cls()
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/__init__.py", line 139, in __init__
    from .wholebody import Wholebody
  File "/workspace/MuseV/controlnet_aux/src/controlnet_aux/dwpose/wholebody.py", line 14, in <module>
    from mmpose.apis import inference_topdown
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/apis/__init__.py", line 2, in <module>
    from .inference import (collect_multi_frames, inference_bottomup,
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/apis/inference.py", line 17, in <module>
    from mmpose.models.builder import build_pose_estimator
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/__init__.py", line 8, in <module>
    from .heads import *  # noqa
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/heads/__init__.py", line 11, in <module>
    from .transformer_heads import EDPoseHead
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/heads/transformer_heads/__init__.py", line 2, in <module>
    from .edpose_head import EDPoseHead
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmpose/models/heads/transformer_heads/edpose_head.py", line 14, in <module>
    from mmcv.ops import MultiScaleDeformableAttention
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/__init__.py", line 3, in <module>
    from .active_rotated_filter import active_rotated_filter
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
    ext_module = ext_loader.load_ext(
  File "/opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/opt/conda/envs/musev/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEENS6_INS5_12MemoryFormatEEE
(musev) root@2f82e80bc1d7:/workspace/MuseV#

It says there's an undefined symbol ImportError: /opt/conda/envs/musev/lib/python3.10/site-packages/mmcv/_ext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEENS6_INS5_12MemoryFormatEEE -- which is a bit confusing.

I'm running on Windows 11 and tried running docker on WSL2 (ubuntu) and powershell and terminal, but I get the same error regardless.

edit: might be a torch mmcv compatibility issue? here: https://github.com/open-mmlab/mmdetection3d/issues/1152 here: https://blog.csdn.net/fengbao24/article/details/129656052

running: conda install -c conda-forge mmcv-full to see if it resolves it, will update soon.

update: that didn't work, took forever, and also didn't work. too many incompatible packages. trying: conda install -c conda-forge mmcv==2.0.1 but, so far it seems it's going to be the same. Maybe I need to downgrade torch from 2.2 to 2.1?

xzqjack commented 4 months ago

@rmasiso it seems the compatibility issues between windows and mmlab packages.

You can try create a new conda env, install and test them by following link. https://mmdetection.readthedocs.io/en/latest/get_started.html

TMElyralab / MuseV

mmcm issues despite being installed properly (running through docker) #44