[NeurIPS 2023] Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Offical implementation of the NeurIPS 2023 paper: Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models.

Project Page: https://diff-foley.github.io

To-Do:

• Evaluation Tool ☑️
• Stage1 CAVP Training Code ☑️
• Stage2 LDM Training Code ☑️
• Environment Setting
• Diff-Foley Inference Code ☑️
• Diff-Foley Pretrained Model ☑️

News

• (🔥New) 2023/11/5 Diff-Foley Inference Pipeline is released! See the 'Inference Usages'.
• (🔥New) 2023/11/5 Diff-Foley Pretrained Model is released! Download from Hugging Face 🤗 here.
• Including: Stage1-CAVP, Stage2-LDM, Double-Guidance Classifier !!

Inference Usages:

1. Open the diff_foley_inference.ipynb in inference folder.
2. Download the pretrained model foler diff_foley_ckpt from Hugging Face 🤗 here and place it under inference folder.
3. Run the diff_foley_inference.ipynb.

Diff-Foley

BibTeX

@misc{luo2023difffoley, 
title={Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models}, 
author={Simian Luo and Chuanhao Yan and Chenxu Hu and Hang Zhao}, 
year={2023}, 
eprint={2306.17203}, 
archivePrefix={arXiv}, 
primaryClass={cs.SD} 
}