MuAViC

A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation.

Paper

Overview

MuAViC provides

• 1200 hours of transcribed audio-visual speech for 9 languages (English, Arabic, German, Greek, Spanish, French, Italian, Portuguese and Russian)
• text translations for 6 English-to-X directions and 6 X-to-English directions (X = Greek, Spanish, French, Italian, Portuguese or Russian)

The raw data is collected from TED/TEDx talk recordings.

Detailed statistics

Getting Data

We provide scripts to generate the audio/video data and AV-HuBERT training manifests for MuAViC.

First, clone this repo for the scripts

git clone https://github.com/facebookresearch/muavic.git

Install required packages:

conda install -c conda-forge ffmpeg==4.2.2
conda install -c conda-forge sox
pip install -r requirements.txt

Then get audio-visual speech recognition and translation data via

python get_data.py --root-path ${ROOT} --src-lang ${SRC_LANG}

where the speech language ${SRC_LANG} is one of en, ar, de, el, es, fr, it, pt and ru.

Generated data will be saved to ${ROOT}/muavic:

• ${ROOT}/muavic/${SRC_LANG}/audio for processed audio files
• ${ROOT}/muavic/${SRC_LANG}/video for processed video files
• ${ROOT}/muavic/${SRC_LANG}/*.tsv for AV-HuBERT AVSR training manifests
• ${ROOT}/muavic/${SRC_LANG}/${TGT_LANG}/*.tsv for AV-HuBERT AVST training manifests

Models

In the following table, we provide all end-to-end trained models mentioned in our paper:

Demo

To try out our state-of-the-art audio-visual models with different audio and video inputs, including a recorded video through the webcam or an uploaded video, checkout our demo:

https://github.com/facebookresearch/muavic/assets/15960959/d03df3b0-488c-443c-ba3b-452b1a5765d8

You can read more about our model in the README file in the demo folder.

Training

For training Audio-Visual models, we are going to use AV-HuBERT framework.

1. Clone and install AV-HuBERT in the root directory:

   $ # Clone the "muavic" branch of av_hubert's repo
   $ git -b muavic clone https://github.com/facebookresearch/av_hubert.git
   $ # Set the fairseq version
   $ cd avhubert
   $ git submodule init
   $ git submodule update
   $ # Install av-hubert's requirements
   $ pip install -r requirements.txt
   $ # Install fairseq
   $ cd fairseq
   $ pip install --editable ./

2. Download an AV-HuBERT pre-trained model from here.

3. Open the training script (scripts/train.sh) and replace these variables:

   # language direction (e.g "en" or "en-fr")
   LANG=
   
   # path where output trained models will be located
   OUT_PATH= 
   
   # path to the downloaded pre-trained model
   PRETRAINED_MODEL_PATH=

4. Run the training script:

   $ bash scripts/train.sh

Note:\ All audio-visual models found here used the large_vox_iter5.pt pre-trained model.

Decoding/Evaluating

To evaluate your trained model (or our [trained models]()) against MuAViC, follow these steps:

1. Open the decoding script (scripts/decode.sh) and replace these variables:

   # language direction (e.g "en" or "en-fr")
   LANG=???
   
   # data split (e.g "test" or "valid")
   GROUP=???
   
   # inference modality (choices: "audio", "video", "audio,video")
   MODALITIES=???
   
   # path to the trained model
   MODEL_PATH=???
   
   # path where decoding results and scores will be located
   OUT_PATH=???

2. Run the decoding script:

   $ bash scripts/decode.sh

   License

CC-BY-NC 4.0

Citation

@article{anwar2023muavic,
  title={MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation},
  author={Anwar, Mohamed and Shi, Bowen and Goswami, Vedanuj and Hsu, Wei-Ning and Pino, Juan and Wang, Changhan},
  journal={arXiv preprint arXiv:2303.00628},
  year={2023}
}