+STARTUP: showeverything

** whisper.el

Speech-to-Text interface for Emacs using OpenAI's [[https://github.com/openai/whisper][whisper speech recognition model]]. For the inference engine it uses the awesome C/C++ port [[https://github.com/ggerganov/whisper.cpp][whisper.cpp]] that can run on consumer grade CPU (without requiring a high end GPU).

You can capture audio with your local input device (microphone) or choose a media file on disk in your local language, and have the transcribed text pasted to your Emacs buffer (optionally after translating to English). This runs offline without having to use non-free cloud service for decent result (though result quality of whisper varies widely depending on language, see below).

*** Install and Usage

Aside from a C++ compiler (to compile whisper.cpp), the system needs to have =FFmpeg= for recording audio.

You can install =whisper.el= by cloning this repo somewhere, and then use it like:

+begin_src elisp

(use-package whisper :load-path "path/to/whisper.el" :bind ("C-H-r" . whisper-run) :config (setq whisper-install-directory "/tmp/" whisper-model "base" whisper-language "en" whisper-translate nil whisper-use-threads (/ (num-processors) 2)))

+end_src

You will use these functions:

• =whisper-run=: Toggle between recording from your microphone and transcribing
• =whisper-file=: Same as before but transcribes a local file on disk

Invoking =whisper-run= with a prefix argument (C-u) has the same effect as =whisper-file=.

Both of these functions will automatically compile whisper.cpp dependency and download language model the first time they are run. When recording is in progress, invoking them stops it and starts transcribing. Otherwise if compilation, download (of model file) or transcription job is in progress, calling them again cancels that.

Note for MacOS users: If whisper.el is failing silently, it might be because Emacs doesn't have the permission to use the Mic. Follow the [[https://github.com/natrys/whisper.el/wiki/MacOS-Configuration#grant-emacs-permission-to-use-mic][recipe]] in wiki to grant it explicitly.

*** Variables

• =whisper-install-directory=: Location where whisper.cpp will be installed. Default is =~/.emacs.d/.cache/=.
• =whisper-language=: Specify your spoken language. Default is =en=. For all possible short-codes: [[https://github.com/ggerganov/whisper.cpp/blob/aa6adda26e1ee9843dddb013890e3312bee52cfe/whisper.cpp#L31][see here]]. You can also set it to =auto= to allow whisper.cpp to infer the language from first 30 seconds of audio. How well whisper works will vary depending on the language. Some scores could be found in the original paper, or [[https://github.com/openai/whisper#available-models-and-languages][here]].
• =whisper-model=: Which language model to use. Default is =base=. Values are: tiny, base, small, medium, large-v1, large. Bigger models are more accurate, but takes more time and more RAM to run (aside from more disk space and download size), see: [[https://github.com/ggerganov/whisper.cpp#memory-usage][resource requirements]]. Note that these come with .en variants that might be faster, but are for English only.
• =whisper-translate=: Default =nil= means transcription output language is same as spoken language. Setting it to =t= translates it to English first.
• =whisper-insert-text-at-point=: By default whisper.el inserts the transcribed text at the point where =whisper-run= or =whisper-file= was invoked. But if you set this to =nil=, the text will be displayed in a separate buffer instead.
• =whisper-return-cursor-to-start=: By default when whisper.el is inserting transcribed text at the point where =whisper-run= is invoked, it keeps cursor at original point to better denote where transcribed text begins. Set this to =nil= to let cursor go to the end of the transcribed text.
• =whisper-use-threads=: Default =nil= means let whisper.cpp choose appropriate value (which it sets with formula min(4, num_of_cores)). If you want to use more than 4 threads (as you have more than 4 cpu cores), set this number manually.
• =whisper-recording-timeout=: Default is =300= seconds. We do not want to start recording and then forget. The intermediate temporary file is stored in uncompressed =wav= format (roughly 4.5mb per minute but can vary), they can grow and fill disk even if ~/tmp/~ is used for it by default.
• =whisper-show-progress-in-mode-line=: By default, progress level of running job in whisper.cpp is shown in the mode line.
• =whisper-quantize=: Whether to quantize the model (default =nil=). Non-nil valid values are: q4_0, q4_1, q5_0, q5_1, q8_0. For an explanation of what quantization means, [[https://github.com/natrys/whisper.el#quantize-the-model][see below]]. If it's defined, whisper.el will automatically quantize the model before using that.
• =whisper-install-whispercpp=: By default the installation of whisper.cpp is done automatically. If you are on a platform where our automatic install fails, but you are able to do so manually at =whisper-install-directory=, you can set this to ='manual= to ensure we don't try and fail to install it automatically. Also if you are planning to not use whisper.cpp at all by overriding =whisper-command= ([[https://github.com/natrys/whisper.el#use-something-other-than-whispercpp][see below]]), you can just set this to =nil= to ensure no whisper.cpp related runtime checks and downloads will be performed.

Additionally, depending on your input device and system you may need to modify these variables to get recording to work:

• =whisper--ffmpeg-input-format=: This is what you would pass to the =-f= flag of FFmpeg to input to record audio. Default is =pulse= on Linux, =avfoundation= on OSX and =dshow= on Windows.
• =whisper--ffmpeg-input-device=: This is what you would pass to the =-i= flag of FFmpeg to record audio, like ~hw:0,2~ or something. If you are using pulseaudio or pipewire-pulse in linux, then the default is =default= source, otherwise this will likely need to be set. For MacOS users, the wiki contains a recipe that lets you set this interactively: [[https://github.com/natrys/whisper.el/wiki/MacOS-Configuration#what-should-be-the-value-of-whisper--ffmpeg-input-device][see here]].

Pulseaudio and PipeWire users who haven't further configured their =default= source may find that results are better when at least =echo cancel= filter is enabled, by loading relevant module. Then you could either set that as the default source (using e.g. =pactl=) or just use that source's name in =whisper--ffmpeg-input-device=. Furthermore, the following programs could be used to improve quality of audio recording in general:

• [[https://github.com/wwmm/easyeffects][Easy Effects]]
• [[https://github.com/noisetorch/NoiseTorch][NoiseTorch]]
• [[https://github.com/Rikorose/DeepFilterNet][DeepFilterNet]] (see also [[https://github.com/Rikorose/DeepFilterNet/blob/main/ladspa/README.md][PipeWire integration]])

*** Hooks

There are a few hooks provided for registering user defined actions:

• =whisper-before-transcription-hook=: Functions here are run before anything else. Helpful to ensure suitable condition to run whisper (e.g. check if buffer is read-only).
• =whisper-after-transcription-hook=: If you want to do some text formatting or transformation on the whisper output, add a function here. Each function would run in a temporary buffer containing transcription output, with point set to beginning of the buffer. For example, the default command output is one big line of text. If you want to do something like adding a paragraph break every N sentences, you could do:

  +begin_src elisp

  (defun whisper--break-sentences (n) "Put a paragraph break every N sentences." (catch 'return (while t (dotimes (_ n) (forward-sentence 1) (when (eobp) (throw 'return nil))) (insert "\n\n") (when (= (char-after) ?\ ) (delete-horizontal-space)))))

(add-hook 'whisper-post-process-hook (lambda () (whisper--break-sentences 5))) ;; add a paragraph break every 5 sentences

+end_src

• =whisper-after-insert-hook=: These functions are run after transcription is completed and the text has been inserted into the buffer. For example, you can pipe the output into another function:

  +begin_src elisp

  (defun pipe-transcribed-audio-to-foo () "Pipe whisper's transcription output into `foo'." (let ((transcription (buffer-substring (line-beginning-position) (line-end-position)))) (foo transcription)))

(add-hook 'whisper-after-insert-hook

'pipe-transcribed-audio-to-foo)

+end_src

*** Performance Guide for Advanced Users

By default, whisper.cpp performance on CPU is likely good enough for most people and most use cases. However if it's not good enough for you, here are some things you could do:

**** Update whisper.cpp

The upstream whisper.cpp is continuously improving. If you are using an old version, updating whisper.cpp is the first thing you could try. Simplest way to do that is to delete your whisper.cpp installation folder and re-run the command, which will reinstall from latest commit.

**** Quantize the model

Quantization is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types. This sacrifices precision for resource efficiency. The idea is that quantized version of a bigger model may afford you to use it (if you are RAM constrained e.g.) with some penalty or accuracy, while still being more accurate hopefully than the smaller model you would be using otherwise.

**** Re-compile whisper.cpp for hardware acceleration

Offloading the encoder inference to hardware or optimised external libraries may result in speed-up. There are options to use: Core ML (for Apple hardware), cuBLAS (for NVIDIA GPU), OpenVINO (Intel CPU/GPU), CLBlast (for GPUs that support OpenCL), OpenBLAS (an optimised matrix processing library for CPU). Consult [[https://github.com/ggerganov/whisper.cpp][whisper.cpp README]] for how to re-compile whisper.cpp to enable those.

**** Use something other than whisper.cpp

If you think there is something else you want to use, you have the option to override the =whisper-command= function definition, or define an overriding advice. This function takes a path to input audio file as argument, and returns a list denoting the command (compatible to =:command= argument to [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Asynchronous-Processes.html][make-process]]), to be run instead of whisper.cpp. You can use the variables described above in this readme to devise the command. The wiki [[https://github.com/natrys/whisper.el/wiki/Setup-to-use-whisper%E2%80%90ctranslate2-instead-of-whisper.cpp][contains a recipe]] that shows how to use [[https://github.com/Softcatala/whisper-ctranslate2][whisper-ctranslate2]] with whisper.el. This client is compatible to OpenAI's original one, so porting the recipe to use the original client should be possible.

Note that when you are using something other than whisper.cpp, the onus is on you to make sure the target program is properly installed and relevant model files for it are downloaded beforehand. We don't support anything other than whisper.cpp so any problems integrating them with whisper.el that's specific to those software may strain our ability to address.

*** Caveats

• Whisper is open-source in the sense that weights and the engine source is available. But training data or methodology is not.
• Real time transcribing is probably not feasible with it yet. The accuracy is better when it has a bigger window of surrounding context. Plus it would need beefy hardware to keep up, possibly using a smaller model. There is some interesting activity going on at whisper.cpp upstream, but in the end I don't see the appeal of that in my workflow (yet).