audio.whisper

This repository contains an R package which is an Rcpp wrapper around the whisper.cpp C++ library.

• The package allows to transcribe audio files using the "Whisper" Automatic Speech Recognition model
• The package is based on an inference engine written in C++11, no external software is needed, so that you can directly install and use it from R

Available models

Installation

For the stable version of this package:

• remotes::install_github("bnosac/audio.whisper", ref = "0.3.3") (uses whisper.cpp version 1.5.4)
• remotes::install_github("bnosac/audio.whisper", ref = "0.2.2") (uses whisper.cpp version 1.2.1)

Look to the documentation of the functions: help(package = "audio.whisper")

• For the development version of this package: remotes::install_github("bnosac/audio.whisper")
• Once you gain familiarity with the flow, you can gain faster transcription speeds by reading this section.

Example

Load the model either by providing the full path to the model or specify the shorthand which will download the model

• see the help of whisper_download_model for a list of available models and to download a model
• you can always download the model manually at https://huggingface.co/ggerganov/whisper.cpp

library(audio.whisper)
model <- whisper("tiny")
model <- whisper("base")
model <- whisper("small")
model <- whisper("medium")
model <- whisper("large-v1")
model <- whisper("large-v2")
model <- whisper("large-v3")
path  <- system.file(package = "audio.whisper", "repo", "ggml-tiny.en-q5_1.bin")
model <- whisper(path)

• If you have a GPU (e.g. Mac with Metal or Linux with CUDA and installed audio.whisper as indicated below), you can use it by specifying: model <- whisper("medium", use_gpu = TRUE), otherwise you will use your CPU.

Transcribe a .wav audio file

• using predict(model, "path/to/audio/file.wav") and
• provide a language which the audio file is in (e.g. en, nl, fr, de, es, zh, ru, jp)
• the result contains the segments and the tokens

audio <- system.file(package = "audio.whisper", "samples", "jfk.wav")
trans <- predict(model, newdata = audio, language = "en", n_threads = 2)

trans
$n_segments
[1] 1

$data
 segment         from           to                                                                                                       text
       1 00:00:00.000 00:00:11.000  And so my fellow Americans ask not what your country can do for you ask what you can do for your country.

$tokens
 segment      token token_prob
       1        And  0.7476438
       1         so  0.9042299
       1         my  0.6872202
       1     fellow  0.9984470
       1  Americans  0.9589157
       1        ask  0.2573057
       1        not  0.7678108
       1       what  0.6542882
       1       your  0.9386917
       1    country  0.9854987
       1        can  0.9813995
       1         do  0.9937403
       1        for  0.9791515
       1        you  0.9925495
       1        ask  0.3058807
       1       what  0.8303462
       1        you  0.9735528
       1        can  0.9711444
       1         do  0.9616748
       1        for  0.9778513
       1       your  0.9604713
       1    country  0.9923630
       1          .  0.4983074

Format of the audio

Note about that the audio file needs to be a 16000Hz 16-bit .wav file.

• you can use R package av which provides bindings to ffmpeg to convert to that format
• or alternatively, use ffmpeg as follows: ffmpeg -i input.wmv -ar 16000 -ac 1 -c:a pcm_s16le output.wav

library(av)
download.file(url = "https://www.ubu.com/media/sound/dec_francis/Dec-Francis-E_rant1.mp3", 
              destfile = "rant1.mp3", mode = "wb")
av_audio_convert("rant1.mp3", output = "output.wav", format = "wav", sample_rate = 16000)

Notes on silences

If you want remove silences from your audio files. You could use R packages

• audio.vadwebrtc
• audio.vadsilero

Speed of transcribing

The tensor operations contained in ggml.h / ggml.c are highly optimised depending on the hardware of your CPU

• It has AVX intrinsics support for x86 architectures, VSX intrinsics support for POWER architectures, Mixed F16 / F32 precision, for Apple silicon allows optimisation via Arm Neon, the Accelerate framework and Metal and provides GPU support for NVIDIA
• In order to gain from these massive transcription speedups, you need to set the correct compilation flags when you install the R package, otherwise transcription speed will be suboptimal (a 5-minute audio fragment can either be transcribed in 40 minutes or 10 seconds depending on your hardware).
• Normally using the installation as described above, some of these compilation flags are detected and you'll see these printed when doing the installation
• It is however advised to set these compilation C flags yourself as follows right before you install the package such that /src/Makevars knows you want these optimisations for sure. This can be done by defining the environment variables WHISPER_CFLAGS, WHISPER_CPPFLAGS, WHISPER_LIBS as follows.

Sys.setenv(WHISPER_CFLAGS = "-mavx -mavx2 -mfma -mf16c")
remotes::install_github("bnosac/audio.whisper", ref = "0.3.3", force = TRUE)
Sys.unsetenv("WHISPER_CFLAGS")

To find out which hardware acceleration options your hardware supports, you can go to https://github.com/bnosac/audio.whisper/issues/26 and look for the CFLAGS (and optionally CXXFLAGS and LDFLAGS) settings which make sense on your hardware

• Common settings to set for WHISPER_CFLAGS are -mavx -mavx2 -mfma -mf16c and extra possible flags -msse3 and mssse3
  • E.g. on my local Windows Intel machine I could set -mavx -mavx2 -mfma -mf16c
  • For Mac users you can speed up transcriptions by setting before installation of audio.whisper
    • Sys.setenv(WHISPER_ACCELERATE = "1") if your computer has the Accelerate framework
    • Sys.setenv(WHISPER_METAL = "1") if your computer has a GPU based on Metal
  • For Linux users which have a NVIDIA GPU, processing can be offloaded to the GPU to a large extent through cuBLAS. For this speedup, install the R package with following settings
    • Sys.setenv(WHISPER_CUBLAS = "1")
    • make sure nvcc is in the PATH (e.g. export PATH=/usr/local/cuda-12.3/bin${PATH:+:${PATH}}) and set the path to CUDA if it is not at /usr/local/cuda as in Sys.setenv(CUDA_PATH = "/usr/local/cuda-12.3")
  • On my older local Ubuntu machine there were no optimisation possibilities. Your mileage may vary.
  • If you have OpenBLAS installed, you can considerably speed up transcription by installing the R package with Sys.setenv(WHISPER_OPENBLAS = "1")
• If you need extra settings in PKG_CPPFLAGS (CXXFLAGS), you can e.g. use Sys.setenv(WHISPER_CPPFLAGS = "-mcpu=native") before installing the package
• If you need extra settings in PKG_LIBS, you can e.g. use Sys.setenv(WHISPER_LIBS = "-framework Accelerate") before installing the package
• If you need custom settings, you can update PKG_CFLAGS / PKG_CPPFLAGS / PKG_LIBS in /src/Makevars directly.

Note that if your hardware does not support these compilation flags, you'll get a crash when transcribing audio.

Support in text mining

Need support in text mining? Contact BNOSAC: http://www.bnosac.be