CrisperWhisper is an advanced variant of OpenAI's Whisper, designed for fast, precise, and verbatim speech recognition with accurate (crisp) word-level timestamps. Unlike the original Whisper, which tends to omit disfluencies and follows more of a intended transcription style, CrisperWhisper aims to transcribe every spoken word exactly as it is, including fillers, pauses, stutters and false starts.
Audio | Whisper Large V3 | Crisper Whisper |
---|---|---|
Demo de 1 | Er war kein Genie, aber doch ein fähiger Ingenieur. | Es ist zwar kein. Er ist zwar kein Genie, aber doch ein fähiger Ingenieur. |
Demo de 2 | Leider müssen wir in diesen schweren Zeiten auch unserem Tagesgeschäft nachgehen. Der hier vorgelegte Kulturhaushalt der Ampelregierung strebt an, den Erfolgskurs der Union zumindest fiskalisch fortzuführen. | Leider [UH] müssen wir in diesen [UH] schweren Zeiten auch [UH] unserem [UH] Tagesgeschäft nachgehen. Der hier [UH] vorgelegte [UH] Kulturhaushalt der [UH] Ampelregierung strebt an, den [UH] Erfolgskurs der Union [UH] zumindest [UH] fiskalisch fortzuführen. Es. |
Demo de 3 | die über alle FRA-Fraktionen hinweg gut im Blick behalten sollten, auch weil sie teilweise sehr teeteuer sind. Aber nicht nur, weil sie teeteuer sind. Wir steigen mit diesem Endentwurf ein in die sogenannten Pandemie-Bereitschaftsverträge. | Die über alle Fr Fraktionen hinweg gut im [UH] Blick behalten sollten, auch weil sie teil teilweise sehr te teuer sind. Aber nicht nur, weil sie te teuer sind. Wir [UH] steigen mit diesem Ent Entwurf ein in die sogenannten Pand Pandemiebereitschaftsverträge. |
Demo en 1 | alternative is you can get like, you have those Dr. Bronner's | Alternative is you can get like [UH] you have those, you know, those doctor Brahmer's. |
Demo en 2 | influence our natural surrounding? How does it influence our ecosystem? | Influence our [UM] our [UH] our natural surrounding. How does it influence our ecosystem? |
Demo en 3 | and always find a place on the street to park and it was easy and you weren't a long distance away from wherever it was that you were trying to go. So I remember that being a lot of fun and easy to do and there were nice places to go and good events to attend. Come downtown and you had the Warner Theater and | And always find a place on the street to park. And and it was it was easy and you weren't a long distance away from wherever it was that you were trying to go. So, I I I remember that being a lot of fun and easy to do and there were nice places to go and, [UM] i good events to attend. Come downtown and you had the Warner Theater and, [UM] |
Demo en 4 | you know, more masculine, who were rough, and that definitely wasn't me. Then, you know, I was very smart because my father made sure I was smart, you know. So, you know, I hung around those people, you know. And then you had the ones that were just out doing things that they shouldn't have been doing also. So, yeah, I was in the little geek squad. You were in the little geek squad. Yeah. | you know, more masculine, who were rough, and that definitely wasn't me. Then, you know, I was very smart because my father made sure I was smart. You know, so, [UM] you know, I I hung around those people, you know. And then you had the ones that were just just out doing things that they shouldn't have been doing also. So yeah, I was the l I was in the little geek squad. Do you |
CrisperWhisper significantly outperforms Whisper Large v3, especially on datasets that have a more verbatim transcription style in the ground truth, such as AMI and TED-LIUM.
Dataset | CrisperWhisper | Whisper Large v3 |
---|---|---|
AMI | 8.72 | 16.01 |
Earnings22 | 12.37 | 11.3 |
GigaSpeech | 10.27 | 10.02 |
LibriSpeech clean | 1.74 | 2.03 |
LibriSpeech other | 3.97 | 3.91 |
SPGISpeech | 2.71 | 2.95 |
TED-LIUM | 3.35 | 3.9 |
VoxPopuli | 8.61 | 9.52 |
CommonVoice | 8.19 | 9.67 |
Average WER | 6.66 | 7.7 |
CrisperWhisper demonstrates superior performance segmentation performance. This performance gap is especially pronounced around disfluencies and pauses. The following table uses the metrics as defined in the paper. For this table we used a collar of 50ms. Heads for each Model were selected using the method described in the How section and the result attaining the highest F1 Score was choosen for each model using varying number of heads.
Dataset | Metric | CrisperWhisper | Whisper Large v2 | Whisper Large v3 |
---|---|---|---|---|
AMI IHM | F1 Score | 0.79 | 0.63 | 0.66 |
Avg IOU | 0.67 | 0.54 | 0.53 | |
Common Voice | F1 Score | 0.80 | 0.42 | 0.48 |
Avg IOU | 0.70 | 0.32 | 0.43 | |
TIMIT | F1 Score | 0.69 | 0.40 | 0.54 |
Avg IOU | 0.56 | 0.32 | 0.43 |
More plots and ablations can be found in the run_experiments/plots
folder.
Clone the Repository:
git clone https://github.com/nyrahealth/CrisperWhisper.git
cd CrisperWhisper
Create Python Environment:
conda create --name crisperWhisper python=3.10
conda activate crisperWhisper
Install Dependencies:
pip install -r requirements.txt
Additional Installations:
Follow OpenAI's instructions to install additional dependencies like ffmpeg
and rust
: Whisper Setup.
Here's how to use CrisperWhisper in your Python scripts:
First make sure that you have a huggingface account and accept the licensing of the model. Grab you huggingface access token and login so you are certainly able to download the model.
huggingface-cli login
import os
import sys
import torch
from datasets import load_dataset
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from utils import adjust_pauses_for_hf_pipeline_output
device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
model_id = "nyrahealth/CrisperWhisper"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)
processor = AutoProcessor.from_pretrained(model_id)
pipe = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
chunk_length_s=30,
batch_size=16,
return_timestamps='word',
torch_dtype=torch_dtype,
device=device,
)
dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]
hf_pipeline_output = pipe(sample)
crisper_whisper_result = adjust_pauses_for_hf_pipeline_output(hf_pipeline_output)
print(crisper_whisper_result)
We also provide a converted model to be compatible with faster whisper. However, due to the different implementation of the timestamp calculation in faster whisper or more precisely CTranslate2 the timestamp accuracy can not be guaranteed.
First make sure that you have a huggingface account and accept the licensing of the model. Grab you huggingface access token and login so you are certainly able to download the model.
huggingface-cli login
from faster_whisper import WhisperModel
from datasets import load_dataset
faster_whisper_model = 'nyrahealth/faster_CrisperWhisper'
# Initialize the Whisper model
device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = "float16" if torch.cuda.is_available() else "float32"
model = WhisperModel(faster_whisper_model, device=device, compute_type="float32")
dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]
segments, info = model.transcribe(sample['array'], beam_size=1, language='en', word_timestamps = True, without_timestamps= True)
for segment in segments:
print(segment)
First make sure that you have a huggingface account and accept the licensing of the model. Grab you huggingface access token and login so you are certainly able to download the model.
huggingface-cli login
afterwards:
To transcribe an audio file, use the following command:
python transcribe.py --f <path_to_audio_file>
To use the CrisperWhisper model with a user-friendly interface, you can run the provided Streamlit app. This app allows you to record or upload audio files for transcription and view the results with accurate word-level timestamps.
Make sure you have followed the Setup ⚙️ instructions above and have the crisperWhisper
environment activated.
Activate the Conda Environment
Ensure you are in the crisperWhisper
environment:
conda activate crisperWhisper
Navigate to the App Directory
Change directory to where the app.py
script is located:
Run the Streamlit App
Use the following command to run the app. Make sure to replace /path/to/your/model
with the actual path to your CrisperWhisper model directory:
streamlit run app.py -- --model_id /path/to/your/model
For example:
streamlit run app.py -- --model_id nyrahealth/CrisperWhisper
Access the App
After running the command, the Streamlit server will start, and you can access the app in your web browser at:
http://localhost:8501
We employ the popular Dynamic Time Warping (DTW) on the Whisper cross-attention scores, as detailed in our paper to derive word-level timestamps. By leveraging our retokenization process, this method allows us to consistently detect pauses. Given that the accuracy of the timestamps heavily depends on the DTW cost matrix and, consequently, on the quality of the cross-attentions, we developed a specialized loss function for the selected alignment heads to enhance precision.
Although this loss function was not included in the original paper due to time constraints preventing the completion of experiments and training before the submission deadline, it has been used to train our publicly available models. Key Features of this loss are as follows:
Data Preparation
Token-Word Alignment
Ground Truth Cross-Attention
Loss Calculation
1 - cosine similarity
between the predicted cross-attention vector (when predicting a token) and the ground truth cross-attention vector.5 Alignment Head selection
Shield: [![CC BY-NC 4.0][cc-by-nc-shield]][cc-by-nc]
This work is licensed under a
[Creative Commons Attribution-NonCommercial 4.0 International License][cc-by-nc].
[![CC BY-NC 4.0][cc-by-nc-image]][cc-by-nc]
[cc-by-nc]: https://creativecommons.org/licenses/by-nc/4.0/
[cc-by-nc-image]: https://licensebuttons.net/l/by-nc/4.0/88x31.png
[cc-by-nc-shield]: https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg