search

deepset-ai / haystack

AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.73k stars 1.92k forks source link

ONNX Conversion for deepset/deberta-v3-large-squad2 #5527

Closed ss2342 closed 1 year ago

ss2342 commented 1 year ago

I am trying to convert the deberta model to onnx for faster inference but got the following exception:

Exception: The current ONNX conversion only support 'BERT', 'RoBERTa', and 'XLMRoberta' models.

It would be worthwhile to try to have onnx support for Deberta given how much it outperforms the rest of the models on squad.

anakin87 commented 1 year ago

Hello, @ssingh1997!

The ONNX conversion part in Haystack is not being actively developed. It is based on the Hugging Face Transformers library.

Doing a quick online search, I found that you can easily convert Transformers models to ONNX using the Hugging Face Optimum library: see this guide.

I hope it helps...

Timoeller commented 1 year ago

Closing as wont do.