search

MyNiuuu / MOFA-Video

Official Pytorch implementation for MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model.
https://myniuuu.github.io/MOFA_Video
Other
343 stars 21 forks source link

how to install? #16

Closed 4lt3r3go closed 21 hours ago

4lt3r3go commented 3 days ago

i would appreciate a simple guide on how to install this. wasted 2 hours already trying everything, git clone and stuff.

MyNiuuu commented 3 days ago

Thank you for your interest! We have updated the instructions here. Please follow these steps to install the demo. Feel free to ask questions if you encounter any specific issues.

4lt3r3go commented 3 days ago

thank for doing a "noob proof" guide. however this part is not clear, there are tons of files over there. should i download file by file, everything inside each folders and keep each folder names and structure? image

MyNiuuu commented 3 days ago

Hi! You may use git lfs to download the entire ckpts folder. Here's how:

  1. Download git lfs from https://git-lfs.github.com. It is commonly used for cloning repositories with large model checkpoints on HuggingFace.
  2. Execute git clone https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid to download the complete HuggingFace repository, which currently only includes the ckpts folder.
  3. Copy or move the ckpts folder to the GitHub repository.

NOTE: If you encounter the error git: 'lfs' is not a git command on Linux, you can try this solution that has worked well for my case.

4lt3r3go commented 3 days ago

thank you for the patience and quick answer, this was a solution i saw but i was scared to try since never did this. here is what i did (wich is part of the installation instruction) created a folder then git clone https://github.com/MyNiuuu/MOFA-Video.git then cd ./MOFA-Video then conda create -n mofa python==3.10

at this point i stopped and reading this message. Now i dont know in wich folder should i clone all those ckpts. sorry. Should i create some other folder and then run that command above inside the new folder? i mean this command: git clone https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid

MyNiuuu commented 3 days ago

You can make a temporary download folder and then remove it once done. For example, after executing the following commands:

git clone https://github.com/MyNiuuu/MOFA-Video.git
cd ./MOFA-Video
conda create -n mofa python==3.10

Assuming you're on a Linux system, you can execute the below commands (ensure that git lfs is installed first from https://git-lfs.github.com/):

cd ./MOFA-Video-Hybrid
mkdir temp_dir
cd ./temp_dir
git clone https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid
mv MOFA-Video-Hybrid/ckpts ..
cd ..
rm -rf ./temp_dir

By the way, I was able to respond quickly because I was just watching the Euro Cup 2024 before going to bed😂. I might not be able to respond in the next couple of hours because I'll be sleeping (my current time zone is GMT+8).

4lt3r3go commented 3 days ago

(just want to point you everything i'm doing, so you will be able to eventually correct the install guide and no one else will annoy you)

  1. i'm on windows at the moment, and will try use this on windows. i did the git clone https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid inside a temporary folder i created manually. I guess i'll have to figure where to paste/move everything after finish

no idea were to paste precisely that immense amount of stuff in that folder you wrote "the checkpoints should be orgnized as ./MOFA-Video-Hybrid/ckpt_tree.md. " what does that mean?

2.

image

this step gave me this error:

(mofa) PS G:\ai\mofa\MOFA-Video> pip install -r requirements.txt
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

i found 3 different "requirements.txt" in 3 different folders, none was in the one explained in the guide. wich one i should run?

  1. image i had to create this folder, wasn't there.

MyNiuuu commented 3 days ago

i'm on windows at the moment, and will try use this on windows. i did the git clone https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid inside a temporary folder i created manually. I guess i'll have to figure where to paste/move everything after finish no idea were to paste precisely that immense amount of stuff in that folder you wrote "the checkpoints should be orgnized as ./MOFA-Video-Hybrid/ckpt_tree.md. " what does that mean?

You need to paste the ckpts folder into ./MOFA-Video-Hybrid. ./MOFA-Video-Hybrid/ckpt_tree.md is just for checking all the files are correctly downloaded, they will be automatically organized if you use git lfs to download the huggingface repo and copy the ckpts folder into ./MOFA-Video-Hybrid.

(mofa) PS G:\ai\mofa\MOFA-Video> pip install -r requirements.txt ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt' i found 3 different "requirements.txt" in 3 different folders, none was in the one explained in the guide. wich one i should run?

Sorry for the mistake, I missed the cd ./MOFA-Video-Hybrid in the first row. The correct commands would be:

cd ./MOFA-Video-Hybrid
conda create -n mofa python==3.10
conda activate mofa
pip install -r requirements.txt
pip install opencv-python-headless
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

i had to create this folder, wasn't there.

I took the CMP model down by mistake last night. I have now re-uploaded it to the initial location, it can be downloaded now. You can try again.

4lt3r3go commented 2 days ago

i'm sorry, i think i've done all correctly but won't load.

at \mofa-video-hybrid> python run_gradio_video_driven.py i'm getting this:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "G:\ai\mofa\mofa-video\mofa-video-hybrid\run_gradio_video_driven.py", line 29, in <module>
    from pipeline.pipeline import FlowControlNetPipeline
  File "G:\ai\mofa\mofa-video\mofa-video-hybrid\pipeline\pipeline.py", line 19, in <module>
    from models.traj_ctrlnet import FlowControlNet as DragControlNet
  File "G:\ai\mofa\mofa-video\mofa-video-hybrid\models\traj_ctrlnet.py", line 13, in <module>
    from models.softsplat import softsplat
  File "G:\ai\mofa\mofa-video\mofa-video-hybrid\models\softsplat.py", line 4, in <module>
    import cupy
  File "C:\Users\P\miniconda3\envs\mofa\lib\site-packages\cupy\__init__.py", line 20, in <module>
    raise ImportError(f'''
ImportError:
================================================================
Failed to import CuPy.

same error when launching \mofa-video-hybrid> python run_gradio_audio_driven.py

same error when launching the other thing \mofa-video-traj> python run_gradio.py

Traceback (most recent call last):
  File "G:\ai\mofa\mofa-video\mofa-video-traj\run_gradio.py", line 655, in <module>
    DragNUWA_net = Drag("cuda:0", target_size, target_size, 25)
  File "G:\ai\mofa\mofa-video\mofa-video-traj\run_gradio.py", line 225, in __init__
    self.pipeline, self.cmp = init_models(
  File "G:\ai\mofa\mofa-video\mofa-video-traj\run_gradio.py", line 93, in init_models
    from pipeline.pipeline import FlowControlNetPipeline
  File "G:\ai\mofa\mofa-video\mofa-video-traj\pipeline\pipeline.py", line 9, in <module>
    from models.svdxt_featureflow_forward_controlnet_s2d_fixcmp_norefine import FlowControlNet
  File "G:\ai\mofa\mofa-video\mofa-video-traj\models\svdxt_featureflow_forward_controlnet_s2d_fixcmp_norefine.py", line 12, in <module>
    from models.softsplat import softsplat
  File "G:\ai\mofa\mofa-video\mofa-video-traj\models\softsplat.py", line 4, in <module>
    import cupy
  File "C:\Users\P\miniconda3\envs\mofa\lib\site-packages\cupy\__init__.py", line 20, in <module>
    raise ImportError(f'''
ImportError:
================================================================
Failed to import CuPy.

If you installed CuPy via wheels (cupy-cudaXXX or cupy-rocm-X-X), make sure that the package matches with the version of CUDA or ROCm installed.
MyNiuuu commented 2 days ago

What is your CUDA version? It is recommended to install version 11.7.

4lt3r3go commented 2 days ago

image

i hope changing it to 11.7 wont break my other Ai stuff, like forge/auto1111/comfy

MyNiuuu commented 2 days ago

There is an option, you can just change your cupy version to 12.1. So you don't have to change your CUDA version.

But you may have to change the code according to this issue.

You can uninstall cupy117, then install cupy version to 12.1.

Here are the instructions for installing cupy. I think pip install cupy-cuda121 will work for you.

4lt3r3go commented 2 days ago

thanks.
I don't want to be nitpicky or annoy you, I'm just sharing everything that's happening to me so that it can be helpful to anyone else who might be trying to install this

so, since my cuda version is 12.1 i did what you suggested:

pip install cupy-cuda121 gave me this error : ERROR: Could not find a version that satisfies the requirement cupy-cuda121 (from versions: none) ERROR: No matching distribution found for cupy-cuda121

pip install cupy-cuda12x did the job

next

at python run_gradio_video_driven.py now i have this error:

G:\AI\MOFA\MOFA-Video\MOFA-Video-Hybrid>python run_gradio_video_driven.py
C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'C:\Users\****\AppData\Local\Programs\Python\Python310\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.3.0+cu121 with CUDA 1201 (you have 2.0.1+cpu)
    Python  3.10.11 (you have 3.10.9)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
Traceback (most recent call last):
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 710, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\autoencoder_kl_temporal_decoder.py", line 23, in <module>
    from .attention_processor import CROSS_ATTENTION_PROCESSORS, AttentionProcessor, AttnProcessor
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\attention_processor.py", line 31, in <module>
    import xformers
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\xformers\__init__.py", line 12, in <module>
    from .checkpoint import (  # noqa: E402, F401
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\xformers\checkpoint.py", line 464, in <module>
    class SelectiveCheckpointWrapper(ActivationWrapper):
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\xformers\checkpoint.py", line 481, in SelectiveCheckpointWrapper
    @torch.compiler.disable
AttributeError: module 'torch' has no attribute 'compiler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "G:\AI\MOFA\MOFA-Video\MOFA-Video-Hybrid\run_gradio_video_driven.py", line 25, in <module>
    from diffusers import AutoencoderKLTemporalDecoder
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 701, in __getattr__
    value = getattr(module, name)
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 700, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "C:\Users\****\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 712, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import diffusers.models.autoencoder_kl_temporal_decoder because of the following error (look up to see its traceback):
module 'torch' has no attribute 'compiler'
MyNiuuu commented 2 days ago

WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.3.0+cu121 with CUDA 1201 (you have 2.0.1+cpu) Python 3.10.11 (you have 3.10.9)

It seems that your PyTorch are 2.0.1 + cpu? Maybe you have to reinstall other dependencies such as pytorch and diffusers?

If 2.0.1 does not work for you, you can try 2.3.0 instead, I guess.

4lt3r3go commented 2 days ago

here's what i did, on anaconda (miniconda) console:

pip install --upgrade torch torchvision then got an incompatibili error for xformers so i did pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121

got same errors so i tryed to re run pip install -r -requirements.txt

now this:

G:\AI\MOFA\MOFA-Video\MOFA-Video-Hybrid>python run_gradio_video_driven.py
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.3.0+cu121 with CUDA 1201 (you have 2.0.1+cpu)
    Python  3.10.11 (you have 3.10.9)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
Traceback (most recent call last):
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 710, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\autoencoder_kl_temporal_decoder.py", line 23, in <module>
    from .attention_processor import CROSS_ATTENTION_PROCESSORS, AttentionProcessor, AttnProcessor
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\models\attention_processor.py", line 31, in <module>
    import xformers
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\xformers\__init__.py", line 12, in <module>
    from .checkpoint import (  # noqa: E402, F401
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\xformers\checkpoint.py", line 464, in <module>
    class SelectiveCheckpointWrapper(ActivationWrapper):
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\xformers\checkpoint.py", line 481, in SelectiveCheckpointWrapper
    @torch.compiler.disable
AttributeError: module 'torch' has no attribute 'compiler'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "G:\AI\MOFA\MOFA-Video\MOFA-Video-Hybrid\run_gradio_video_driven.py", line 25, in <module>
    from diffusers import AutoencoderKLTemporalDecoder
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 701, in __getattr__
    value = getattr(module, name)
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 700, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "C:\Users\12345\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\utils\import_utils.py", line 712, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import diffusers.models.autoencoder_kl_temporal_decoder because of the following error (look up to see its traceback):
module 'torch' has no attribute 'compiler'

i feel like i'm going to give up its day 2

MyNiuuu commented 2 days ago

You can try to delete xformers. Actually it is not necessary for this demo (it is a package for acceleration, and it is totally okay not to use it). It seems that all errors originate from the incompatibily of xformers.

theoldsong commented 2 days ago

Let's downgrade Cuda to 11.7. I initially used 12.1, but I also encountered the same mistake as you. Later, when it was changed to 11.8, it was found that cupy did not have a supported version of 11.8. In the end, it was downgraded to 11.7.

4lt3r3go commented 2 days ago

downgrade Cuda to 11.7

after 2 days i was there to give up.. but you gave me last hope. can you please tell me how to downgrade correctly? (hopefully wont break my other ai stuff..)

laishujie commented 1 day ago

downgrade Cuda to 11.7

after 2 days i was there to give up.. but you gave me last hope. can you please tell me how to downgrade correctly? (hopefully wont break my other ai stuff..)

My CUDA version is 12.1.

1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.

To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.

Now, the program should run correctly.

4lt3r3go commented 1 day ago

downgrade Cuda to 11.7

after 2 days i was there to give up.. but you gave me last hope. can you please tell me how to downgrade correctly? (hopefully wont break my other ai stuff..)

My CUDA version is 12.1.

1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.

To modify compile_with_cache function:

In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.

Now, the program should run correctly.

thanks but half of this is like alien language for me. I just give up, wasted 3 days. thanks for your time