How to use ONNX model?

[H-G-11]

Hello there,

I'm interested in using the ONNX model, as I saw that you are providing the weights for it. I tried to use it with optimum library, but didn't manage to make it work. Could someone indicate in which direction I should look into?

Thank you so much for this repository and the work you put into it. It really helps!!

Note:

here is what I tried

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch
from optimum.onnxruntime import ORTModelForSpeechSeq2Seq

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "distil-whisper/distil-large-v2"

model = ORTModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype,  encoder_file_name=f"encoder_model.onnx"
)

Here is the error:

RuntimeError: Too many ONNX model files were found in distil-whisper/distil-large-v2, specify which one to load by using the encoder_file_name argument.

[csukuangfj]

@HuguesGallier

I suggest that you give a look at http://github.com/k2-fsa/sherpa-onnx

It supports both distil-whisper and openai-whisper.

For instance, you can find a colab notebook below about how to run distil-whisper onnx models with sherpa-onnx.

---

Note that you can use sherpa-onnx on Windows/macOS/Linux and it also supports Android/iOS/Rapsberry Pi, etc.

[H-G-11]

Thank you so much, I will have a look at it!