search

arianneorpilla / jidoujisho

A full-featured immersion language learning suite for mobile.
GNU General Public License v3.0
911 stars 58 forks source link

Generate subtitles with whisper #162

Open mathewthe2 opened 1 year ago

mathewthe2 commented 1 year ago

Generate raw subtitles with whisper or align existing by fuzzy matching / recursive search. Alternatively, align them with alass and expose with flutter_rust_bridge.

Since the whisper models can be dynamically loaded, we can exclude them from the build and fetch from huggingface when needed.

arianneorpilla commented 1 year ago

Hi @mathewthe2, might I ask if you know how performant these solutions are? Is this functionality that I can execute with very little time on a mobile device?

I haven't used these before but I'm aware this has been very useful for some people on desktop.

My current advice to people that want to realign their subtitles is to realign and process on their desktop before using on my app but it would be nice to have this be automated by the app itself. But similar to #83, if it's something that I can't manage to do in convenient time on mobile, I'm afraid I won't be able to do this.

mathewthe2 commented 1 year ago

I have only tested it on a mobile emulator on a Mac with the rust binding, but it works pretty well.

I have tested it on 5 minute and 30 minute audio files. It may take anywhere from 30 seconds to 5 minutes, which is a reasonable wait time imo for a tv episode or short audio book.