search

KoljaB / RealtimeSTT

A robust, efficient, low-latency speech-to-text library with advanced voice activity detection, wake word activation and instant transcription.
MIT License
2.09k stars 190 forks source link

translate not working for transcribe method of faster-whisper #114

Open bilalshafim opened 2 months ago

bilalshafim commented 2 months ago

I have added the translate task like so in the _transcription_worker. 

  segments, info = model.transcribe(
      audio,
      language=language if language else None,
      beam_size=beam_size,
      initial_prompt=initial_prompt,
      suppress_tokens=suppress_tokens,
      task="translate"
  )

Is there a reason why it does not translate or could this be a faster-whisper issue? I'll try to setup faster-whisper separately and test it there as well.

Tested with: German, Japanese on small model.

KoljaB commented 2 months ago

Where did you add this code? Do you really mean "translate" or transcribe (faster whisper can also translate but it is not the main use case). How do you see that it is not doing what you expected?

bilalshafim commented 2 months ago

I have edited my previous comment now and added the resolution in this comment if anyone else stumbles upon this.

My test setup: Server 1- Websocket server with RealtimeSTT (for transcribing) Server 2- Exactly identical as Server 1, with the exception of the translate task added to model.transcribe. (running on another GPU - for translating to English)

Resolution: If using real time transcription, add the translate task there as well like so:

  segments, info = self.realtime_model_type.transcribe(
      audio_array,
      language=self.language if self.language else None,
      beam_size=self.beam_size_realtime,
      initial_prompt=self.initial_prompt,
      suppress_tokens=self.suppress_tokens,
      task="translate"
  )

(Sorry I was really tired at that time and missed some things. Guess that's what happens after working straight for 11 hours.)

I can create a PR with the option of adding a 'task' parameter to AudioToTextRecorder class. I want to integrate this into an app where the user would have an option for toggling translation for STT.