search

Dadangdut33 / Speech-Translate

A realtime speech transcription and translation application using Whisper OpenAI and free translation API. Interface made using Tkinter. Code written fully in Python.
MIT License
423 stars 55 forks source link

[BUG] The constant YouTube phrases when used for live subtitles #71

Closed k566o closed 4 months ago

k566o commented 5 months ago

Just wondering why I get constant YouTuber phrases when watching live streams that have nothing to do with what is said. Listening to a Chinese stream right now which isn't on YT, in the past 5 minutes I have got:

Please feel free to like, subscribe, forward, and reward to support the Der Spiegel and Diandian columns. Thank you for watching. Thank you for following and see you again. Video Editing: Seo Jeong-hyuk Der Spiegel needs your support. Welcome to subscribe to Der Spiegel. Please feel free to like, subscribe, forward, and reward to support the Der Spiegel and Diandian columns. Thank you for watching!

I do not get this in other whisper apps. Is there a way you can make this stop, or a way for users to filter out such YouTube phrases?

The Chinese live stream has a girl talking to camera about the national park she is streaming in and there is background crowd voices/noise. This also happens with Korean streams I watch

Thanks.

k566o commented 5 months ago

This happens with with Large V3 model and Medium model. I have not used other models

AdvancedAI-NL commented 5 months ago

It has to do with the hallucination of STT engines and what they were trained on. I assume that whisper was trained on a lot of videos and hence the phrases like thanks for watching, whenever the model thinks it hears speech it will for some reason choose a hallucinated phrase as the output, but fortunately these are mostly filtered out. If you look into the python sitepackages folder and then go to the speech_translate folder, you will find the hallucination filters. There are multiple, but in the folder you should go to the _user folder and then you have 2 hallucination json files. Also if you go back to the main folder and then go to the assets folder, you will find the base_hallucination_filter.json file. I reckon if you add the entries you are constantly encountering, that it will go away or at least become less. Changing these files is at your own risk of course.

k566o commented 5 months ago

Thanks for explanation and possible fix

Dadangdut33 commented 4 months ago

It's exactly as @AdvancedAI-NL explained. You can update the filter to help counter that problem