This fork was created to provide a convenient web interface for using Hallo. The original code has been slightly modified to allow for more control over the generation process.
⚠️ To run the web interface, you need at least 12 GB of video memory (VRAM) and more than 12 GB of RAM. ⚠️
Unfortunately, I was unable to create a free tier Colab notebook as there is not enough RAM available.
But you can try if you have pro
colab
If you have windows and you don't want to bother with installing libs, you can download the portable version , unpack and launch run.bat
docker compose up -d
this will start the gradio web ui and expose the port 7680 which is mapped to 8020 to teh container's host The app will be available at http://localhost:8020
Note : Be sure to use the correct cuda start image for your GPU driver version, if it doesn't build from the start.
Clone this repository:
git clone https://github.com/yourusername/hallo.git
Run install.bat
to set up the environment and download the pretrained models.
Make sure ffmpeg is installed on your system. It doesn't matter where it's located, as long as the system can find it.
Launch the web interface by running start.bat
.
Clone this repository:
git clone https://github.com/yourusername/hallo.git
Run install.sh
to set up the environment and download the pretrained models.
Ensure ffmpeg is installed on your system. You can install it with:
sudo apt-get install ffmpeg
Launch the web interface by running start.sh
.
If you prefer to install manually, here are the detailed steps:
Clone the repository and pretrained models:
git lfs install
git clone https://github.com/yourusername/hallo.git
git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models
curl -L -o pretrained_models/hallo/net.pth https://huggingface.co/fudan-generative-ai/hallo/resolve/main/hallo/net.pth?download=true
Create a virtual environment and activate it:
python -m venv venv
venv\Scripts\activate # For Windows
source venv/bin/activate # For Linux
Install the required packages:
pip install -r requirements.txt
pip install -e .
pip install bitsandbytes-windows --force-reinstall # For Windows only
Install GPU libraries:
pip install torch==2.2.2+cu121 torchaudio torchvision --index-url https://download.pytorch.org/whl/cu121
pip install onnxruntime-gpu
Launch the web interface:
python app.py
To share , use --share
flag
python app.py --share
2024/06/15
: 🎉🎉🎉 Release the first version on GitHub.2024/06/15
: ✨✨✨ Release some images and audios for inference testing on Huggingface.Create conda environment:
conda create -n hallo python=3.10
conda activate hallo
Install packages with pip
pip install -r requirements.txt
pip install .
Besides, ffmpeg is also need:
apt-get install ffmpeg
The inference entrypoint script is scripts/inference.py
. Before testing your cases, there are two preparations need to be completed:
You can easily get all pretrained models required by inference from our HuggingFace repo.
Clone the the pretrained models into ${PROJECT_ROOT}/pretrained_models
directory by cmd below:
git lfs install
git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models
Or you can download them separately from their source repo:
pretrained_models/face_analysis/models/
. (Thanks to deepinsight)pretrained_models/face_analysis/models
.Finally, these pretrained models should be organized as follows:
./pretrained_models/
|-- audio_separator/
| `-- Kim_Vocal_2.onnx
|-- face_analysis/
| `-- models/
| |-- face_landmarker_v2_with_blendshapes.task # face landmarker model from mediapipe
| |-- 1k3d68.onnx
| |-- 2d106det.onnx
| |-- genderage.onnx
| |-- glintr100.onnx
| `-- scrfd_10g_bnkps.onnx
|-- motion_module/
| `-- mm_sd_v15_v2.ckpt
|-- sd-vae-ft-mse/
| |-- config.json
| `-- diffusion_pytorch_model.safetensors
|-- stable-diffusion-v1-5/
| |-- feature_extractor/
| | `-- preprocessor_config.json
| |-- model_index.json
| |-- unet/
| | |-- config.json
| | `-- diffusion_pytorch_model.safetensors
| `-- v1-inference.yaml
`-- wav2vec/
|-- wav2vec2-base-960h/
| |-- config.json
| |-- feature_extractor_config.json
| |-- model.safetensors
| |-- preprocessor_config.json
| |-- special_tokens_map.json
| |-- tokenizer_config.json
| `-- vocab.json
Hallo has a few simple requirements for input data:
For the source image:
For the driving audio:
We have provided some samples for your reference.
Simply to run the scripts/inference.py
and pass source_image
and driving_audio
as input:
python scripts/inference.py --source_image examples/source_images/1.jpg --driving_audio examples/driving_audios/1.wav
Animation results will be saved as ${PROJECT_ROOT}/.cache/output.mp4
by default. You can pass --output
to specify the output file name. You can find more examples for inference at examples folder.
For more options:
usage: inference.py [-h] [-c CONFIG] [--source_image SOURCE_IMAGE] [--driving_audio DRIVING_AUDIO] [--output OUTPUT] [--pose_weight POSE_WEIGHT]
[--face_weight FACE_WEIGHT] [--lip_weight LIP_WEIGHT] [--face_expand_ratio FACE_EXPAND_RATIO]
options:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
--source_image SOURCE_IMAGE
source image
--driving_audio DRIVING_AUDIO
driving audio
--output OUTPUT output video file name
--pose_weight POSE_WEIGHT
weight of pose
--face_weight FACE_WEIGHT
weight of face
--lip_weight LIP_WEIGHT
weight of lip
--face_expand_ratio FACE_EXPAND_RATIO
face region
Status | Milestone | ETA |
---|---|---|
✅ | Inference source code meet everyone on GitHub | 2024-06-15 |
✅ | Pretrained models on Huggingface | 2024-06-15 |
🚀🚀🚀 | [Traning: data preparation and training scripts]() | 2024-06-25 |
🚀🚀🚀 | [Optimize inference performance in Mandarin]() | TBD |
If you find our work useful for your research, please consider citing the paper:
@misc{xu2024hallo,
title={Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation},
author={Mingwang Xu and Hui Li and Qingkun Su and Hanlin Shang and Liwei Zhang and Ce Liu and Jingdong Wang and Yao Yao and Siyu zhu},
year={2024},
eprint={2406.08801},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Multiple research positions are open at the Generative Vision Lab, Fudan University! Include:
Interested individuals are encouraged to contact us at siyuzhu@fudan.edu.cn for further information.
The development of portrait image animation technologies driven by audio inputs poses social risks, such as the ethical implications of creating realistic portraits that could be misused for deepfakes. To mitigate these risks, it is crucial to establish ethical guidelines and responsible use practices. Privacy and consent concerns also arise from using individuals' images and voices. Addressing these involves transparent data usage policies, informed consent, and safeguarding privacy rights. By addressing these risks and implementing mitigations, the research aims to ensure the responsible and ethical development of this technology.