[DMP 2024]: Auto Subtitler for Indian Languages

arvind-planetread commented 5 months ago

Ticket Contents

As part of the BIRD initiative , we aim to create a tool which can speed up the adoption of Same Language Subtitling (SLS) among the content producers for the entire country. This will ensure that 200M weak readers and 30M readers with accessibility to get regular reading exposure with content having SLS.

This tool will create SRT files by taking a video file and its text file. We aim for the tool to support the following languages : Tamil, Telugu, Kannada for now.

Goals & Mid-Point Milestone

Goal 1: Achieve 60% accuracy in timing accuracy of SRT files in Tamil Language. Achieve 60% accuracy in timing accuracy of SRT files in Telugu Language. Achieve 60% accuracy in timing accuracy of SRT files in Kannada Language.

Goal 2: Achieve 70% accuracy in timing accuracy of SRT files in Tamil Language. Achieve 70% accuracy in timing accuracy of SRT files in Telugu Language. Achieve 70% accuracy in timing accuracy of SRT files in Kannada Language.

Goal 3: Achieve 80% accuracy in timing accuracy of SRT files in Tamil Language. Achieve 80% accuracy in timing accuracy of SRT files in Telugu Language. Achieve 80% accuracy in timing accuracy of SRT files in Kannada Language.

Goal 4: Achieve 90% accuracy in timing accuracy of SRT files in Tamil Language. Achieve 90% accuracy in timing accuracy of SRT files in Telugu Language. Achieve 90% accuracy in timing accuracy of SRT files in Kannada Language.

The midpoint milestones will be completion of Goal 1 and Goal 2.

Setup/Installation

No response

Expected Outcome

The input will be a video file and its script in text file format. The text will be utf8 encoding. The output will be an SRT file with timecode for each line of the script.

Acceptance Criteria

We will use the VLC media player to check the time accuracy of the generated SRT file. This will be used to verify the completion of the goals too. We will use multiple video files to check if the tool is versatile.

Implementation Details

Python or any other technical stack.

Mockups/Wireframes

No response

Product Name

Auto Subtitler for Indian Languages

Organisation Name

Planet Read

Domain

⁠Education

Tech Skills Needed

Machine Learning, Python

Mentor(s)

@arvind-planetread

Weekly Goals

Week 1

[x] Researched various open source model to generate ASR.
[x] Listed out some good models such as OpenAI Whisper, Whisper JAX, Kaldi , Force Alignment and Montreal Force Aligner.
[x] Generated Subtitles using OpenAI Whisper but thr model failed in complex audio files.
[x] Used Montreal Force Aligner for generating subtitles.
[x] Tested and Finalized Montreal Force Alignment for Further Subtitling.
Week 2
[x] Started Training for English Language Subtitling using Montreal Force Aligner tool.
[x] Started Validating the model created with several audio files.
[x] Inferencing on english audios and the out comes were 95 percent accurate.
[x] Started Training on Tamil Language Subtitling using Montreal Force Aligner tool.
[x] Started Validating the model with several tamil audio files.
Week 3
[x] Inferencing on Tamil audios and the out comes were 95 percent accurate.
[x] Started Training for Hindi Language Subtitling using Montreal Force Aligner.
[x] Started Validating the model created with several files.
[x] Inferencing on Hindi audios and the outcomes were 95 percent accurate.
[x] Started Training on Punjabi Language Subtitling using Montreal Force Aligner tool.
Week 4
[x] Started Validating the model created with several audio files.
[x] Upon Inferencing the outcomes are 80 percent accurate.
[x] Created a streamlit App to Showcase to users.
[x] Creating a port forwarding and publishing it into on cloud is going on.
[x] Adding Starting and Ending time into the frontend to see the SRT in a particular time frame.
Week 5
[x] Creating Roman Subtitles for the created models.
[x] Adding perfect breakups in the subtitles.
[x] Fine tuning created subtitles.
[x] go for training kannada language subtitling using Montreal Force Aligner tool.
[x] Start Validating the model created with several files.
Week 6
[x] Inferencing on Kannada language and checking the outcomes.
[x] Start Training for Telugu language Subtitling using Montreal Force Alignment.
[x] Start Validating the model created with several audio files.
[x] Inferencing the outcomes
[x] Create different APIs for Different Languages.
Week 7
[x] Host the APIs in gunicorn and uvicorn like platforms.
[x] Create a simple frontend which is to be used by movie producers.
[x] Start testing on several videos.
[x] keep testing testing and testing.
Week 8
[ ] Create API Docs using FastAPI.
[ ] Use Swagger for Documentation of the APIs.
[ ] Create an entire documentation jupyter notebook file and codes and CLI properly.
Week 9
[ ] Create a Readme to replicate the opensource development done above.
Week 10
[ ] Through testing and bug fixes if any.
Week 11
[ ] Testing and validating the model for 5 hours video duration.
Week 12
[ ] Document for final evaluation.

PlanetRead / PR-Repository