jafrizzell / moonlights-transcribe

0 stars 0 forks source link

Assign unique Whisper model per stream #5

Open jafrizzell opened 1 year ago

jafrizzell commented 1 year ago

Look into effects of transcribing multiple streams on the same loaded model, and consider loading a unique model instance for each stream.

Because the Whisper model uses past transcription results to aid with the current transcription task, mixing streams on a single model will reduce the effectiveness of the model, particularly when the streams are discussing highly different topics.

Downside: Loading two models may increase RAM usage. Server currently idles around ~60% RAM usage (transcription idles at ~750MB).

Side affect: Different streams may produce new Whisper hallucinations, and the transcription cleaning may need to be re-worked

jafrizzell commented 1 year ago

Code-wise this is very easy to implement. Need to look into the affect this will have on the performance of the server.