An open source subtitling platform π» for transcribing videos/audios in Indic languages and translating subtitles as well using ML models.
A. SeamlessM4T model
We are planning to use Meta's Seamless Communication technology which was recently released in github [1]. The SeamlessM4T_v2_large
model π, supports almost 12 Indic languages [2] by default. With this model alone, we can potentially transcribe audio in respective languages and even translate subtitles into other languages. More details about SeamlessM4T can be found in paper [7]. The functionality is very well explained in this tutorial [8] written in Seamless Communication Repository.
In lot of Indic languages, there are fine-tuned Whisper ASR models in respective languages. More such models can be found in this Whisper event leaderboard [3]. We have personally fine-tuned Whisper models in my mother tongue malayalam like [4] and [5]. So if performance of any language is not really good in SeamlessM4T model, we can switch to one of the fine-tuned Whisper ASR based models available in open source or make one ourselves. Yet one thing to note though is, that Whisper might not be able to support all the languages listed in Seamless.
Indic Languages supported with SeamlessM4T
Language | Code |
---|---|
Assamese | asm |
Bengali | ben |
English | eng |
Gujarati | guj |
Hindi | hin |
Kannada | kan |
Malayalam | mal |
Marathi | mar |
Odia | ory |
Punjabi | pan |
Tamil | tam |
Telugu | tel |
Urdu | urd |
The language code abbrevation for each of the models can be found out here [6].
B. faster-whisper
faster-whisper [9] is a reimplementation of OpenAI's Whisper model using CTranslate2, which is a fast inference engine for Transformer models. This implementation is up to 4 times faster than openai/whisper for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU. Since faster-whisper is based in Whisper, it supports all the 99 languages supported by whisper.
Indic Languages supported with faster-whisper
Language | Code |
---|---|
Assamese | as |
Bengali | bn |
English | en |
Gujarati | gu |
Hindi | hi |
Kannada | kn |
Malayalam | ml |
Marathi | mr |
Punjabi | pa |
Tamil | ta |
Telgu | te |
Urdu | ur |
C. WhisperX
WhisperX provides fast automatic speech recognition (70x realtime with large-v2) with word-level timestamps and speaker diarization. The features provided by WhisperX are:
Indic Languages supported with faster-whisper
Language | Code |
---|---|
English | en |
Hindi | hi |
Telgu | te |
Urdu | ur |
D. fine-tuned Whisper model
In certain languages, Whisper by default is not performing strongly. In your problem, the open source Whisper model doesnβt give good results. Then fine-tune your ASR model with examples like Fine-Tune Whisper For Multilingual ASR with π€ Transformers.
Indic Languages supported with fine-tuned Whisper model
Language | Code |
---|---|
Malayalam | ml |
We plan to use FastAPI as the backend and deploy it on serveless platforms like Modal.com or any other alternatives.
API format
generate_seamlessm4t_speech
, generate_faster_whisper_speech
, generate_whisperx_speech
API with the following input format:{
"wav_base64": "Audio in base64 format",
"target": "Your target lanugage you want to transcribe or translate your audio"
}
youtube_generate_seamlessm4t_speech
, youtube_generate_faster_whisper_speech
, youtube_generate_whisperx_speech
API with the following input format:{
"yt_id": "Youtube ID as input in string format",
"target": "Your target lanugage you want to transcribe or translate your audio"
}
Next.js, being a React framework, offers you all the benefits of React plus more features out of the box, such as file-based routing and API routes, which can simplify your development process. It's an excellent choice, especially for a web application that requires server-side rendering (SSR) or static site generation (SSG) for better performance and SEO.
Framework: Next.js (enables SSR and SSG, improving load times and SEO) Styling: Tailwind CSS or styled-components (for styling with ease and efficiency)