search

KoljaB / RealtimeSTT

A robust, efficient, low-latency speech-to-text library with advanced voice activity detection, wake word activation and instant transcription.
MIT License
2.06k stars 184 forks source link
python realtime speech-to-text

readme

RealtimeSTT

PyPI Downloads GitHub release GitHub commits GitHub forks GitHub stars

Easy-to-use, low-latency speech-to-text library for realtime applications

New

About the Project

RealtimeSTT listens to the microphone and transcribes voice into text.

Hint: Check out Linguflex, the original project from which RealtimeSTT is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.

It's ideal for:

https://github.com/user-attachments/assets/797e6552-27cd-41b1-a7f3-e5cbc72094f5

Updates

Latest Version: v0.3.7

See release history.

Hint: Since we use the multiprocessing module now, ensure to include the if __name__ == '__main__': protection in your code to prevent unexpected behavior, especially on platforms like Windows. For a detailed explanation on why this is important, visit the official Python documentation on multiprocessing.

Quick Examples

Print everything being said:

from RealtimeSTT import AudioToTextRecorder
import pyautogui

def process_text(text):
    print(text)

if __name__ == '__main__':
    print("Wait until it says 'speak now'")
    recorder = AudioToTextRecorder()

    while True:
        recorder.text(process_text)

Type everything being said:

from RealtimeSTT import AudioToTextRecorder
import pyautogui

def process_text(text):
    pyautogui.typewrite(text + " ")

if __name__ == '__main__':
    print("Wait until it says 'speak now'")
    recorder = AudioToTextRecorder()

    while True:
        recorder.text(process_text)

Will type everything being said into your selected text box

Features

Hint: Check out RealtimeTTS, the output counterpart of this library, for text-to-voice capabilities. Together, they form a powerful realtime audio wrapper around large language models.

Tech Stack

This library uses:

These components represent the "industry standard" for cutting-edge applications, providing the most modern and effective foundation for building high-end solutions.

Installation

pip install RealtimeSTT

This will install all the necessary dependencies, including a CPU support only version of PyTorch.

Although it is possible to run RealtimeSTT with a CPU installation only (use a small model like "tiny" or "base" in this case) you will get way better experience using:

GPU Support with CUDA (recommended)

Updating PyTorch for CUDA Support

To upgrade your PyTorch installation to enable GPU support with CUDA, follow these instructions based on your specific CUDA version. This is useful if you wish to enhance the performance of RealtimeSTT with CUDA capabilities.

For CUDA 11.8:

To update PyTorch and Torchaudio to support CUDA 11.8, use the following commands:

pip install torch==2.3.1+cu118 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu118

For CUDA 12.X:

To update PyTorch and Torchaudio to support CUDA 12.X, execute the following:

pip install torch==2.3.1+cu121 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

Replace 2.3.1 with the version of PyTorch that matches your system and requirements.

Steps That Might Be Necessary Before

Note: To check if your NVIDIA GPU supports CUDA, visit the official CUDA GPUs list.

If you didn't use CUDA models before, some additional steps might be needed one time before installation. These steps prepare the system for CUDA support and installation of the GPU-optimized installation. This is recommended for those who require better performance and have a compatible NVIDIA GPU. To use RealtimeSTT with GPU support via CUDA please also follow these steps:

  1. Install NVIDIA CUDA Toolkit:

  2. Install NVIDIA cuDNN:

    • select between CUDA 11.8 or CUDA 12.X Toolkit
      • for 12.X visit cuDNN Downloads.
        • Select operating system and version.
        • Download and install the software.
      • for 11.8 visit NVIDIA cuDNN Archive.
        • Click on "Download cuDNN v8.7.0 (November 28th, 2022), for CUDA 11.x".
        • Download and install the software.

  3. Install ffmpeg:

    Note: Installation of ffmpeg might not actually be needed to operate RealtimeSTT *thanks to jgilbert2017 for pointing this out

    You can download an installer for your OS from the ffmpeg Website.

    Or use a package manager:

Quick Start

Basic usage:

Manual Recording

Start and stop of recording are manually triggered.

recorder.start()
recorder.stop()
print(recorder.text())

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

if __name__ == '__main__':
    recorder = AudioToTextRecorder()
    recorder.start()
    input("Press Enter to stop recording...")
    recorder.stop()
    print("Transcription: ", recorder.text())

Automatic Recording

Recording based on voice activity detection.

with AudioToTextRecorder() as recorder:
    print(recorder.text())

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

if __name__ == '__main__':
    with AudioToTextRecorder() as recorder:
        print("Transcription: ", recorder.text())

When running recorder.text in a loop it is recommended to use a callback, allowing the transcription to be run asynchronously:

def process_text(text):
    print (text)

while True:
    recorder.text(process_text)

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

def process_text(text):
    print(text)

if __name__ == '__main__':
    recorder = AudioToTextRecorder()

    while True:
        recorder.text(process_text)

Wakewords

Keyword activation before detecting voice. Write the comma-separated list of your desired activation keywords into the wake_words parameter. You can choose wake words from these list: alexa, americano, blueberry, bumblebee, computer, grapefruits, grasshopper, hey google, hey siri, jarvis, ok google, picovoice, porcupine, terminator. 

recorder = AudioToTextRecorder(wake_words="jarvis")

print('Say "Jarvis" then speak.')
print(recorder.text())

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

if __name__ == '__main__':
    recorder = AudioToTextRecorder(wake_words="jarvis")

    print('Say "Jarvis" to start recording.')
    print(recorder.text())

Callbacks

You can set callback functions to be executed on different events (see Configuration) :

def my_start_callback():
    print("Recording started!")

def my_stop_callback():
    print("Recording stopped!")

recorder = AudioToTextRecorder(on_recording_start=my_start_callback,
                               on_recording_stop=my_stop_callback)

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

def start_callback():
    print("Recording started!")

def stop_callback():
    print("Recording stopped!")

if __name__ == '__main__':
    recorder = AudioToTextRecorder(on_recording_start=start_callback,
                                   on_recording_stop=stop_callback)

Feed chunks

If you don't want to use the local microphone set use_microphone parameter to false and provide raw PCM audiochunks in 16-bit mono (samplerate 16000) with this method:

recorder.feed_audio(audio_chunk)

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

if __name__ == '__main__':
    recorder = AudioToTextRecorder(use_microphone=False)
    with open("audio_chunk.pcm", "rb") as f:
        audio_chunk = f.read()

    recorder.feed_audio(audio_chunk)
    print("Transcription: ", recorder.text())

Shutdown

You can shutdown the recorder safely by using the context manager protocol:

with AudioToTextRecorder() as recorder:
    [...]

Or you can call the shutdown method manually (if using "with" is not feasible):

recorder.shutdown()

Standalone Example:

from RealtimeSTT import AudioToTextRecorder

if __name__ == '__main__':
    with AudioToTextRecorder() as recorder:
        [...]
    # or manually shutdown if "with" is not used
    recorder.shutdown()

Testing the Library

The test subdirectory contains a set of scripts to help you evaluate and understand the capabilities of the RealtimeTTS library.

Test scripts depending on RealtimeTTS library may require you to enter your azure service region within the script. When using OpenAI-, Azure- or Elevenlabs-related demo scripts the API Keys should be provided in the environment variables OPENAI_API_KEY, AZURE_SPEECH_KEY and ELEVENLABS_API_KEY (see RealtimeTTS)

The example_app subdirectory contains a polished user interface application for the OpenAI API based on PyQt5.

Configuration

Initialization Parameters for AudioToTextRecorder

When you initialize the AudioToTextRecorder class, you have various options to customize its behavior.

General Parameters

Real-time Transcription Parameters

Note: When enabling realtime description a GPU installation is strongly advised. Using realtime transcription may create high GPU loads.

Voice Activation Parameters

Wake Word Parameters

OpenWakeWord

Training models

Look here for information about how to train your own OpenWakeWord models. You can use a simple Google Colab notebook for a start or use a more detailed notebook that enables more customization (can produce high quality models, but requires more development experience).

Convert model to ONNX format

You might need to use tf2onnx to convert tensorflow tflite models to onnx format:

pip install -U tf2onnx
python -m tf2onnx.convert --tflite my_model_filename.tflite --output my_model_filename.onnx

Configure RealtimeSTT

Suggested starting parameters for OpenWakeWord usage:

    with AudioToTextRecorder(
        wakeword_backend="oww",
        wake_words_sensitivity=0.35,
        openwakeword_model_paths="word1.onnx,word2.onnx",
        wake_word_buffer_duration=1,
        ) as recorder:

FAQ

Q: I encountered the following error: "Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so} Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor." How do I fix this?

A: This issue arises from a mismatch between the version of ctranslate2 and cuDNN. The ctranslate2 library was updated to version 4.5.0, which uses cuDNN 9.2. There are two ways to resolve this issue:

  1. Downgrade ctranslate2 to version 4.4.0: 
    pip install ctranslate2==4.4.0
  2. Upgrade cuDNN on your system to version 9.2 or above.

Contribution

Contributions are always welcome!

Shoutout to Steven Linn for providing docker support.

License

MIT

Author

Kolja Beigel
Email: kolja.beigel@web.de
GitHub