Open lightbooster opened 7 months ago
@lightbooster Hi, whisper has not supported this option yet. I would update here if it works or when we could remove the 30s restrictions.
@lightbooster 嗨,whisper 尚未支持此选项。如果它有效或者我们可以移除 30 秒的限制,我会在这里更新。
if this is supported now?
if this is supported now?
Currently, for the distill-whispr or fine-tuned Whisper models, it is possible to configure audio other than 30 seconds. The --remove-input-padding option is also supported, but it does not actually remove the padding internally; it only supports the input and output format. Support for arbitrary length audio input has not yet been implemented.
尚未实现对任意长度音频输入的支持。
We are using whipser for streaming speech recognition. Will this padding increase the amount of calculation at the beginning of the audio stream, and will the reasoning affect the speed?
We are using whipser for streaming speech recognition. Will this padding increase the amount of calculation at the beginning of the audio stream, and will the reasoning affect the speed?
It will increase computation, but it won't add too much because a large part of the model's time consumption is determined by the number of autoregressive steps of the decoder. Padding does not increase this number.
By the way, because training and inference must be consistent, the accuracy of the native Whisper model will be compromised if the input is audio other than 30 seconds.
We are using whipser for streaming speech recognition. Will this padding increase the amount of calculation at the beginning of the audio stream, and will the reasoning affect the speed?
It will increase computation, but it won't add too much because a large part of the model's time consumption is determined by the number of autoregressive steps of the decoder. Padding does not increase this number.
By the way, because training and inference must be consistent, the accuracy of the native Whisper model will be compromised if the input is audio other than 30 seconds.
Thanks for the answer. We removed padding during training.
System Info
nvidia/cuda:12.1.0-devel-ubuntu22.04
0.9.0.dev2024020600
Who can help?
@byshiue
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
running
build.py
script fromexamples/whisper
with--remove_input_padding
option:Expected behavior
serialized engine expected, without necessity for padding batch to 30 seconds samples
[batch_size, n_mels, 3000]
actual behavior
actually
build.py
script will be failed with assertion in encoder attention layer:additional notes
tensorrt_llm/layers/attention.py
code snippet:actual shape of
qkv
:without
--remove_input_padding
option, everything works fine and as expected