search

sandrohanea / whisper.net

Whisper.net. Speech to text made simple using Whisper Models
MIT License
591 stars 92 forks source link

The problem of multilingual mixing in recording #177

Open duwenlong2 opened 8 months ago

duwenlong2 commented 8 months ago

I have a scene like this. The audio file for meeting minutes needs to be converted into text, but after using Whisper, there are Chinese and English sentences in my audio file. He has uniformly output English. I want to preserve the results of the original language. Can it be implemented in Whisper?

sandrohanea commented 1 month ago

The library is identifying one language at stat-up (if "auto" is used) and then it is used to transcribe the entire file so it makes sense it will uniformly output English (and transcriptions when other languages are spoken).

One idea to fix it (but it is not tested) would be to :

  1. Add the WithProbabilities on the builder => which will give you the confidence level for each segment.
  2. Once you identify some segment with low confidence level, re-transcribe it (by extracting the frames for that segment from start and end time). Either provide the other language if you know it is always "Chinese and English" or identify it again using "auto".

  3. Replace the segments in the result.

    It would be probably interesting to have this functionality in the library in the future, but cannot promise that I'll have time to implement it.