batching inference and forced decoding for speedup and multi-target

ufal / whisper_streaming

Whisper realtime streaming for long speech-to-text transcription and translation

MIT License

1.88k stars 226 forks source link

batching inference and forced decoding for speedup and multi-target #55

Open Gldkslfmsd opened 8 months ago

Gldkslfmsd commented 8 months ago

Batching inference should be used in Whisper-Streaming. It's currently not implemented.

This could work: https://github.com/huggingface/transformers/pull/27658

if "forced decoding" really works for Whisper, it should help to avoid re-processing the current buffer from start of segment, and it should be faster

Why batching:

If more than chunk-size audio is accumulated, process a batch of the full audio buffer, and the buffer minus chunk size. Then apply local agreement as on the two subsequent iterations. It will be faster.
it could enable joint transcription and translation on one GPU. It might be slower than separately -- due to padding, one of them might have short buffer and the other long. But not so much with forced decoding. And it might be good anyway
it could enable multiple clients in one instance

vidalfer commented 8 months ago

Hi! There's an implementation that supports batch inference: https://github.com/Vaibhavs10/insanely-fast-whisper I'm not sure if it can be easily implemented in the Whisper streaming project

Gldkslfmsd commented 8 months ago

Hi! There's an implementation that supports batch inference: https://github.com/Vaibhavs10/insanely-fast-whisper

I'm not sure if it can be easily implemented in the Whisper streaming project

yes, me neither. I would need a pointer to the function that takes two audio samples and processes them at once.

Gldkslfmsd commented 8 months ago

OK, I checked it. The Insanely Fast Whisper is just a wrapper of Huggingface Transformers. The example usage of batching is https://github.com/huggingface/transformers/pull/27658 .

This https://github.com/pe-trik/transformers/blob/online_decode/examples/pytorch/online-decoding/whisper-online-demo.py shows the forced decoding.

So these are the initial points to work on this issue. I might do it in a few weeks, but anybody can go on :)

joaogabrieljunq commented 7 months ago

OK, I checked it. The Insanely Fast Whisper is just a wrapper of Huggingface Transformers. The example usage of batching is huggingface/transformers#27658 .

This https://github.com/pe-trik/transformers/blob/online_decode/examples/pytorch/online-decoding/whisper-online-demo.py shows the forced decoding.

So these are the initial points to work on this issue. I might do it in a few weeks, but anybody can go on :)

Any news about this implementation or any finds so we can try to work on that? I am trying to build a multi-client server and batching would be nice to run more than one transcript at the same instance

Gldkslfmsd commented 7 months ago

Wow, great! No news, except that batching has becoming also my priority :) Let's cooperate. I want to start later this week. My first step will be a jupyter notebook where I'll quickly inspect and prototype. It will be messy. Then I isolate the working solution into this repo.

The easiest use case for batching is decode the same audio twice, the whole buffer + the whole minus last chunk.

joaogabrieljunq commented 7 months ago

Sure, let's cooperate! My doubt is: decode the same audio twice is for speedup use case, right? I check you mention about multi-client in https://github.com/ufal/whisper_streaming/issues/42 and would it be necessary to decode + batching backend API to parallelize multiple audios in GPU? I could try to work in this batching backend layer using whisper-streaming source code.

Gldkslfmsd commented 7 months ago

Sure, let's cooperate! My doubt is: decode the same audio twice is for speedup use case, right?

yes. Just be aware that batching multiple audios can result in slow down. There will be independent audio buffers of different lengths. You need to pad the audio input to the longest, and the processing time is the same as the longest. So you gain effectiveness, but lose some speed.

Gldkslfmsd commented 7 months ago

So, how's your progress, @joaogabrieljunq ? I found this today: https://github.com/m-bain/whisperX/blob/main/whisperx/asr.py They know how to use batching with faster-whisper. I hope I can reuse this code. And I found that huggingface transformers enable batching with Whisper, but most probably not with word-level timestamps. And they're really necessary with Whisper-Streaming.

joaogabrieljunq commented 7 months ago

Hello again @Gldkslfmsd, nice to know that you are progressing in batch implementation research! I spent yesterday researching also about possible implementations for this. Found WhisperS2T that seems to implement dynamic time length support in batch inference, helping in the pad problem that you mentioned above. Perhaps this could help also. https://github.com/shashikg/WhisperS2T/blob/main/whisper_s2t/backends/ctranslate2/model.py

SalomonKisters commented 6 months ago

Any news on this matter?

orgh0 commented 4 months ago

Any update on batching?

Gldkslfmsd commented 4 months ago

no. Unfortunately it's not among my priorities anymore.