How to load .en variants under recent versions?

[lawik]

The english-specific models are smaller, faster and more effective if you know you are dealing with english. Or so I gather.

Trying the regular Livebook Neural Network Smart cell, editing it and switching in .en on the model doesn't run:

This will fail with weird configuration errors:

{:ok, model_info} = Bumblebee.load_model({:hf, "openai/whisper-tiny.en"})
{:ok, featurizer} = Bumblebee.load_featurizer({:hf, "openai/whisper-tiny.en"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "openai/whisper-tiny.en"})
{:ok, generation_config} = Bumblebee.load_generation_config({:hf, "openai/whisper-tiny.en"})
# [..] rest is unmodified

That will give:

** (RuntimeError) invalid task :transcribe, expected one of: 
    (bumblebee 0.4.2) lib/bumblebee/audio/speech_to_text_whisper.ex:210: Bumblebee.Audio.SpeechToTextWhisper.forced_token_ids/2
    (bumblebee 0.4.2) lib/bumblebee/audio/speech_to_text_whisper.ex:156: Bumblebee.Audio.SpeechToTextWhisper.generate_opts/2
    (bumblebee 0.4.2) lib/bumblebee/audio/speech_to_text_whisper.ex:50: Bumblebee.Audio.SpeechToTextWhisper.speech_to_text_whisper/5
    #cell:6sklelzqlgjf6jghgh7dyemmpp4u7iau:8: (file)

Changing it to this runs but produces an empty transcript:

{:ok, model_info} = Bumblebee.load_model({:hf, "openai/whisper-tiny.en"})
{:ok, featurizer} = Bumblebee.load_featurizer({:hf, "openai/whisper-tiny.en"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "openai/whisper-tiny.en"})
{:ok, generation_config} = Bumblebee.load_generation_config({:hf, "openai/whisper-tiny"})

And the default works fine.

I have used tiny.en and friends before generation_config was added at all.

[lawik]

Looking at small and small.en they seem to lack task_to_id:

https://huggingface.co/openai/whisper-small/blob/main/generation_config.json

vs

https://huggingface.co/openai/whisper-small.en/blob/main/generation_config.json

[lawik]

Something similar: https://github.com/huggingface/transformers/issues/25084

[jonatanklosko]

Looking at openai implementation, for the monolingual model they don't include the task token. So passing task: nil to the serving is the way to go. I added an error that suggests that (1020c752ff8fb9a0738620d5452306da373cddd8).

[lawik]

Thanks!