Try out Whisper-X - Githubissues

Veldhoen commented 3 months ago

Deploy and run WhisperX

greenw0lf commented 3 months ago

Notes:

Doesn't work with Python 3.9 (OpenShift cluster default)
Works with Python 3.12 (installed via miniconda)
Only uses 1 GPU device, cannot use multiple (some users tried to use multiple and measured the performance, but sadly using multiple GPUs with WhisperX does not improve performance. See: https://github.com/m-bain/whisperX/issues/391)
Using batch_size = 16, model performs the fastest observed so far. I will experiment with higher batch_size since the model wasn't using the whole GPU memory available.
Diarization requires a Huggingface token - something to keep in mind
At the moment: Not sure if VAD is used during transcription phase (hypothesis: It is used)
- UPDATE: VAD is used
Word-level timestamp alignment generated using wav2vec 2.0

greenw0lf commented 3 months ago

UPDATE:

batch_size = 24 works the best (91% GPU memory usage, fastest overall)
batch_size = 26 was <1s faster for 1 hr audio file and consumed around 97% GPU memory. Overall it was still slower by <1s than size 24. Very close to running OOM => NOT SUITABLE

Next step: Run WhisperX evaluation using batch_size = 24

beeldengeluid / whisper-asr-worker