search

jptrsn / zip-captions

Open source browser closed-caption generator
https://zipcaptions.app
GNU General Public License v3.0
45 stars 7 forks source link

Teach Zipcaptions new words #129

Open PJTCJessica opened 1 month ago

PJTCJessica commented 1 month ago

Hi there!

I'm a new Zipcaptions user. I work for a synagogue and we are using Zipcaptions to make our religious services more accessible for individuals with hearing loss issues. Is it possible to "teach" Zipcaptions to understand commonly used Hebrew terms and phrases that frequently appear in sermons? If so, how do I do that? At the moment, it doesn't recognize common words like Shabbat, the Jewish day of rest. Our concern is that mistranscriptions of Hebrew terms will be distracting and could cause disruptions. Any thoughts and help would be much appreciated!

question #help wanted

jptrsn commented 1 month ago

Thanks for posting your question! It's always a pleasure to hear about how folks are making communities more accessible.

Unfortunately, custom training an AI model is not yet something Zip Captions can support, but we do have this feature in mind for future development. I'm sorry to hear that it's mis-translating Hebrew words, that's quite a frustration, and a challenge for a computer to be able to switch language contexts as easily as we humans can.

Would you be able to provide an example of audio which results in the incorrect results? If also available, the incorrect captions and their correct alternatives would be immensely helpful. I am personally unable to speak Hebrew, so I am quite limited in what I might be able to test on my own, and to recognize words that are incorrectly represented. A link to a youtube video of a typical service (even just a couple minutes worth) would be greatly helpful, and I'm optimistic you may be able to find and share something.

If the incorrect text result is unique enough, I may be able to build a kind-of work around to improve the results by doing a simple detect-and-replace. However, my best efforts will likely be inadequate, as I will be unable to create and test many unique scenarios which your are likely to encounter in a synagogue.

We are working constantly on improvements, but with only a handful of supporters on Patreon, our resources are quite limited both in time and funds to provide a timely response. I hope that we can deliver a solution to help you continue to be able to include everyone in your services.

PJTCJessica commented 1 month ago

Hi jptrsn!

Thanks for your kind response! I should be able to send you the examples you requested after the Jewish High Holy Days are over.

Best,

jptrsn commented 1 month ago

Wonderful, and I wish you and your community a happy, healthy High Holy Days.

:)

dairytech commented 1 month ago

This is interesting. When I was setting up our system for services this past week I used Shabbat shalom as my test phrase and it recognized them correctly. It didn't capitalize them but I didn't expect that as it doesn't capitalize anything. I couldn't test anything else as I don't know any other Hebrew and I couldn't pronounce them like a native speaker would. Your question presents a very interesting use case for the speech to text engine. I said to my wife last night. It is amazing what AI can do and it is equally amazing what it cannot. I am skeptical of those predicting that AI is going to displace most of us in the work force.