CTC segmentation can be used to find utterance alignments within large audio files.
ctc-segmentation
python package.The CTC segmentation package is not standalone, as it needs a neural network with CTC output. It is integrated in these frameworks:
It can also be used with other frameworks:
With pip
:
pip install ctc-segmentation
From the Arch Linux AUR as python-ctc-segmentation-git
using your favourite AUR helper.
From source:
git clone https://github.com/lumaku/ctc-segmentation
cd ctc-segmentation
cythonize -3 ctc_segmentation/ctc_segmentation_dyn.pyx
python setup.py build
python setup.py install --optimize=1 --skip-build
Character probabilites from each time step are obtained from a CTC-based network. With these, transition probabilities are mapped into a trellis diagram. To account for preambles or unrelated segments in audio files, the transition cost are set to zero for the start-of-sentence or blank token.
Starting from the time step with the highest probability for the last character, backtracking determines the most probable path of characters through all time steps.
As this method generates a probability for each aligned character, a confidence score for each utterance can be derived. For example, if a word within an utterance is missing, this value is low.
The confidence score helps to detect and filter-out bad utterances.
There are several notable parameters to adjust the working of the algorithm that can be found in the class CtcSegmentationParameters
:
Localization: The character set is taken from the model dict, i.e., usually are generated with SentencePiece. An ASR model trained in the corresponding language and character set is needed. For asian languages, no changes to the CTC segmentation parameters should be necessary. One exception: If the character set contains any punctuation characters, "#", or the Greek char "ε", adapt the setting in an instance of CtcSegmentationParameters
in segmentation.py
.
CtcSegmentationParameters
includes a blank character. Copy over the Blank character from the dictionary to the configuration, if in the model dictionary e.g. "\
If replace_spaces_with_blanks
is True, then spaces in the ground truth sequence are replaces by blanks. This option is enabled by default and improves compability with dictionaries with unknown space characters.
min_window_size
: Minimum window size considered for a single utterance. The current default value should be OK in most cases.
To align utterances with longer unkown audio sections between them, use blank_transition_cost_zero
(default: False). With this option, the stay transition in the blank state is free. A transition to the next character is only consumed if the probability to switch is higher. In this way, more time steps can be skipped between utterances. Caution: in combination with replace_spaces_with_blanks == True
, this may lead to misaligned segments.
Directly set the parameter index_duration
to give the corresponding time duration of one CTC output index (in seconds).
Example: For a given sample rate, say, 16kHz, fs=16000
. Then, how many sample points correspond to one ctc output index? In some ASR systems, this can be calculated from the hop length of the windowing times encoder subsampling factor. For example, if the hop length of the frontend windowing is 128, and the subsampling factor in the encoder is 4, totalling 512 sample points for one CTC index. Then index_duration = 512 / 16000
.
Note: In earlier versions, index_duration
was not used and the time stamps were determined from the values of subsampling_factor
and frame_duration_ms
. To derive index_duration
from these values, calculateframe_duration_ms * subsampling_factor / 1000
.
Character probabilities over each L frames are accumulated to calculate the confidence score. The L value can be adapted with with score_min_mean_over_L
. A lower L makes the score more sensitive to error in the transcription, but also errors in the ASR model.
CTC segmentation requires CTC activations of an already trained CTC-based network. Example code can be found in the alignment scripts asr_align.py
of ESPnet 1 or ESpnet 2.
prepare_token_list
from a ground truth sequence that was already coverted to a sequence of tokens (recommended). Alternatively, use prepare_text
on raw text, this method filters characters not in the dictionary and can break longer tokens into smaller tokens.ctc_segmentation
computes character-wise alignments from the CTC log posterior probabilites.determine_utterance_segments
converts char-wise alignments to utterance-wise alignments.Sometimes the ground truth data is not text, but a sequence of tokens, or a list of protein segments.
In this case, use either prepare_token_list
or replace it with a function that suits better for your data.
For examples, see the prepare_*
functions in ctc_segmentation.py
, or the example included in the NeMo toolkit.
Segments that were written to a segments
file can be filtered using the confidence score. This is the minium confidence score in log space as described in the paper.
Utterances with a low confidence score are discarded in a data clean-up. This parameter may need adjustment depending on dataset, ASR model and used text conversion.
min_confidence_score=1.5
awk -v ms=${min_confidence_score} '{ if ($5 > ms) {print} }' ${unfiltered} > ${filtered}
How do I split a large text into multiple utterances? This can be done automatically, e.g. in our paper, the text of this book/chapter was split into utterances at sentence endings to derive utterances.
What if there are unrelated parts within the audio file without transcription? Unrelated segments can be skipped with the gratis_blank
parameter. Larger unrelated segments may deteriorate the results, try to increase the minimum window size. Partially repeating segments have a high chance to disarrange the alignments, remove them if possible. These segments can be detected with the confidence score. Use the state_list
to see how well the unrelated part was "ignored".
How fast is CTC segmentation? On a modern computer (64GB RAM, Nvidia RTX 2080 ti, AMD Ryzen 7 2700X), it takes around 400 ms to align 500s of audio with tokenized text and the default window size (8000~250s). In comparison, inference of CTC posteriors on CPU takes some time from 4s to 20s; GPU inference takes roughly 500 - 1000 ms, but often fails at such long audio files because of excessive memory consumption (tested with Transformer model on Espnet 2; this strongly depends on model architecture and used toolkit). A few factors influence CTC segmentation speed: Window size, length of audio, length of text, how well the text fits to audio and the preprocessing function. Aligning from tokenized text is faster because the alignment with prepare_text
additionally includes transition probabilities from partial tokens; this increases the complexity by the length of the longest token in the character dictionary (including the blank token).
The inference of the model is too slow and uses too much memory. Use RNN-based ASR networks that grow linearly in complexity with longer audio files. Inference complexity increases on long audio files quadratically for Transformer-based architectures. To solve this, it is possible to partition the audio into several parts. CTC segmentation includes an example partitioning function in ctc_segmentation.get_partitions
. Example Code in JtubeSpeech
How can I improve the alignment speed of CTC segmentation? The alignment algorithm is not parallelizable for batch processing, so use a CPU with a good single-thread performance. It's possible to align multiple files in parallel, if the computer has enough temporary memory. The alignment is faster with shorter max token length, if text is aligned - or directly align from a token list.
How do I get word-based alignments instead of full utterance segments? Use an ASR model with character tokens to improve the time-resolution. Then handle each word as single utterance.
How can I improve the accuracy of the generated alignments? Be aware that depending on the ASR performance of network and other factors, CTC activations are not always accurate, sometimes shifted by a few frames. To get a better time resolution, use a dictionary with characters! Also, the prepare_text
function tries to break down long tokens into smaller tokens.
What is the difference between prepare_token_list
and prepare_text
? Explained in examples:
The full paper can be found in the preprint https://arxiv.org/abs/2007.09127 or published at https://doi.org/10.1007/978-3-030-60276-5_27. The code used in the paper is archived in https://github.com/cornerfarmer/ctc_segmentation. To cite this work:
@InProceedings{ctcsegmentation,
author="K{\"u}rzinger, Ludwig
and Winkelbauer, Dominik
and Li, Lujun
and Watzel, Tobias
and Rigoll, Gerhard",
editor="Karpov, Alexey
and Potapova, Rodmonga",
title="CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition",
booktitle="Speech and Computer",
year="2020",
publisher="Springer International Publishing",
address="Cham",
pages="267--278",
abstract="Recent end-to-end Automatic Speech Recognition (ASR) systems demonstrated the ability to outperform conventional hybrid DNN/HMM ASR. Aside from architectural improvements in those systems, those models grew in terms of depth, parameters and model capacity. However, these models also require more training data to achieve comparable performance.",
isbn="978-3-030-60276-5"
}