Open pprobst opened 4 months ago
I at least expect the model to learn not to use punctuation from the data itself, even if I do not suppress them explicitly during inference, but it's not what is happening here
During training with this method, the model is not trained to predict the given labels exactly, but rather it's trained to have the same hidden states as the original model. Meaning, if the original model thinks the word "comma" in "hello comma how are you" is unlikely, the finetuned model will also think it's unlikely. This is in contrast to regular finetuning, where the finetuned model gets nudged to believe the training data is probable.
It's useful to think of this finetuning process more as a distillation process, where the training data just so happens to be a reference for how the models behave, rather than a source of truth that the model is trained to match. If you want your model to have specific behavior/domain-knowledge and be able to use audio_ctx, you'll have to finetune them normally first, and then use this method to make it more robust with audio_ctx.
I'm not entirely sure about the suppress_tokens issue you mention though
I at least expect the model to learn not to use punctuation from the data itself, even if I do not suppress them explicitly during inference, but it's not what is happening here
During training with this method, the model is not trained to predict the given labels exactly, but rather it's trained to have the same hidden states as the original model. Meaning, if the original model thinks the word "comma" in "hello comma how are you" is unlikely, the finetuned model will also think it's unlikely. This is in contrast to regular finetuning, where the finetuned model gets nudged to believe the training data is probable.
It's useful to think of this finetuning process more as a distillation process, where the training data just so happens to be a reference for how the models behave, rather than a source of truth that the model is trained to match. If you want your model to have specific behavior/domain-knowledge and be able to use audio_ctx, you'll have to finetune them normally first, and then use this method to make it more robust with audio_ctx.
I'm not entirely sure about the suppress_tokens issue you mention though
Oh, this changes everything, thanks. I believe I misunderstood, then. I suppose that if I use my already fine-tuned model and "distil" it using your code, then things will work. I'll do this and post the results here for history.
Hm. I've not been so successful. I ran whispercpp's main
with varying audio context on a benchmark dataset, and these are the results:
ac1500 (default model)
eval_spanish_base-20240328_6k16b_1500/log_2024-04-12T12:59.txt
2024-04-12 13:02:24 __main__ INFO WER: 0.06779981114258735 (6.78%)
ac768 (default model)
eval_spanish_base-20240328_6k16b_768/log_2024-04-12T13:03.txt
9:2024-04-12 13:05:53 __main__ INFO WER: 0.07034938621340887 (7.03%)
# Below are whisper-actf models (fine-tuned over my "default" model over 8 epochs):
ac1500
actf_spanish_base-20240328_6k16b_1500/log_2024-04-12T12:35.txt
2024-04-12 12:38:43 __main__ INFO WER: 0.15401322001888573 (15.40%)
ac768
actf_spanish_base-20240328_6k16b_768/log_2024-04-12T12:43.txt
2024-04-12 12:46:00 __main__ INFO WER: 0.1474032105760151 (14.74%)
ac512
actf_spanish_base-20240328_6k16b_512/log_2024-04-12T12:46.txt
2024-04-12 12:48:34 __main__ INFO WER: 0.1457979225684608 (14.58%)
ac512 (greedy)
actf_spanish_base-20240328_6k16b_512_greedy/log_2024-04-12T12:54.txt
2024-04-12 12:56:32 __main__ INFO WER: 0.15920679886685551 (15.92%)
ac500
actf_spanish_base-20240328_6k16b_500/log_2024-04-12T12:51.txt
2024-04-12 12:54:08 __main__ INFO WER: 0.14513692162417374 (14.51%)
ac256
actf_spanish_base-20240328_6k16b_256/log_2024-04-12T12:49.txt
2024-04-12 12:51:28 __main__ INFO WER: 0.42360717658168084 (42.36%)
WER doubles essentially, even if I use the default ac
.
Just to confirm, you made sure to replace both model_train
and model_base
with copies of your finetuned model? What if you try finetuning with the default dataset instead of the custom one?
Hello. First of all, thank you for your work.
I came upon this repo when trying to improve the transcription speed in whispercpp by using a lower
audio_ctx
. However, while fine-tuning with this code, it does not seem to be suppressing tokens. I adapted the code a bit to setsuppress_tokens
and to use my dataset in Spanish, but everything else of importance remained unchanged.When fine-tuning normally, at least for
WhisperForConditionalGeneration
, settingmodel.config.suppress_tokens = suppress
works. But I'm not sure it's working here. Furthermore, my training dataset (about 6100 audio files) does not use any punctuation marks, that is, "," is "comma" and not the actual character. So I at least expect the model to learn not to use punctuation from the data itself, even if I do not suppress them explicitly during inference, but it's not what is happening here -- even if I train for 15 epochs -- the same number of epochs I use in my "normal" pipeline that does not use dynamic audio context.Also, in the code below, suppressing tokens only works during inference if I set these in the
model.generate
function:This made me suspect that it's not suppressing tokens during training. But I don't know how to verify this.
Full code (except loading the dataset, but nothing unusual there):
Finally, when running inference on a test set using whispercpp and the fine-tuned model, I get a WER of 39% instead of the usual 7% that I get when trained "normally".