afaligner is a Python library for automatic text and audio synchronization (a.k.a. forced aligner). You give it a list of text files and a list of audio files that contain the narrated text, and it produces a mapping between text fragments and the corresponding audio fragments.
afaligner is used in the syncabook command-line tool to produce EPUB3 with Media Overlays ebooks and has been developed for this specific purpose. If you want to create an ebook with synchronized text and audio, consider using syncabook instead of using afaligner directly.
afaligner works by synthesizing text and then aligning synthesized and recorded audio using a variation of the DTW (Dynamic Time Warping) algorithm. The main features of the algorithm are:
It can handle structural differences in the beginning and in the end of files, which is often the case with audiobooks (e.g. disclaimers).
It finds an approximation to an optimal warping path in linear time and space using the FastDTW approach. This and the fact that the algorithm is implemented in C make it pretty fast compared to other forced aligners.
It can, with varying success, align differently split text and audio.
afaligner was inspired by aeneas and works in a similar way. It uses aeneas as a dependency for text synthesis and MFCC extraction.
afaligner works on 64-bit Mac OS and Linux. Windows is not currently supported (you may try to use a VM).
aeneas
, numpy
, jinja2
Install Python (>= 3.6)
Install numpy:
pip install numpy
Install afaligner:
pip install afaligner
Or if you want to modify the afaligner's source code:
Get the repository:
git clone https://github.com/r4victor/afaligner/ && cd afaligner
Install afaligner in editable mode:
pip install -e .
Install pytest
:
pip install pytest
Run tests:
python -m pytest tests/
Installing all the afaligner's dependencies can be tedious, so the library comes with Dockerfile. You can use it to build a Debian-based Docker image that contains afaligner itself and all its dependencies. Alternatively, you can use Dockerfile as a reference to install afaligner on your machine.
Installation via Docker:
Get the repository:
git clone https://github.com/r4victor/afaligner/ && cd afaligner
Build an image:
docker build -t afaligner .
Now you can run the container like so:
docker run -ti afaligner
It enters bash. You can run Python and import afaligner
. To do something useful, you may need to mount your code that uses afaligner as a volume.
afaligner provides only one function called align()
that takes a text directory, an audio directory, and a set of output parameters and returns a sync map (a mapping from text fragments to their time positions in audio files). If the output directory is specified, it also writes the result in the JSON or SMIL format to that directory. The call may look like this:
from afaligner import align
sync_map = align(
'ebooks/demoebook/text/',
'ebooks/demoebook/audio/',
output_dir='ebooks/demoebook/smil/',
output_format='smil',
sync_map_text_path_prefix='../text/',
sync_map_audio_path_prefix='../audio/'
)
and sync_map
has the following structure:
{
"p001.xhtml": {
"f001": {
"audio_file": "p001.mp3",
"begin_time": "0:00:00.000",
"end_time": "0:00:02.600",
},
"f002": {
"audio_file": "p001.mp3",
"begin_time": "0:00:02.600",
"end_time": "0:00:05.880",
},
# ...
},
"p002.xhtml": {
"f016": {
"audio_file": "p002.mp3",
"begin_time": "0:00:00.000",
"end_time": "0:00:03.040",
}
# ...
},
}
For more details, please refer to docstrings.
pip install afaligner
may not work on macOS if it tries to compile a universal library. This seems to be because aeneas
complies only on x86_64. I got an error when using Python 3.9. The following command fixes it:
ARCHFLAGS="-arch x86_64" pip install afaligner