Speculative Decoding - Githubissues

SYSTRAN / faster-whisper

Faster Whisper transcription with CTranslate2

MIT License

12.2k stars 1.02k forks source link

Speculative Decoding #771

Closed RohitMidha23 closed 7 months ago

RohitMidha23 commented 7 months ago

Is speculative decoding faster than faster-whisper?
Is there going to be support anytime soon for speculative decoding in faster-whisper?

Both of these questions are asked with a purely realtime, streaming audio view in mind!

trungkienbkhn commented 7 months ago

@RohitMidha23 , hello.

I tested speculative decoding for whisper from here, then compared with fw large-v2. I used an audio file 192s and RTX 3090 GPU card. Below are the transcription times:

fw large-v2: 18.649s
HF speculative decoding: 17.394s => It can be seen that speculative decoding support from HF transformers has improved the time a bit, but not too much.

Yes we will look into this

RohitMidha23 commented 7 months ago

Thanks for those results! Looking forward to seeing speculative decoding in Faster Whisper!

S-Cardenas commented 4 months ago

Is there any update on adding speculative decoding to Faster Whisper? If there is an open ticket or pull request I'd be willing to help and contribute to it.