search

chainyo / transformers-pipeline-onnx

How to export Hugging Face's 🤗 NLP Transformers models to ONNX and use the exported model with the appropriate Transformers pipeline.
23 stars 0 forks source link

Saving pipeline #2

Closed alunap closed 1 year ago

alunap commented 2 years ago

First, thank you very much for this!

There is one final piece that I suggest adding to your notebook, which is to save the pipeline itself to ONNX. I haven't seen this done anywhere for NLP, but I gather it is possible to save a pipeline and not just a model.

The great thing about that is that any device can then load the pipeline and put in text directly. As it is, the hugging face model itself gets saved and so can be run from, say C#, but it also needs the transformer, and the hugging face transformers are available with Rust, Python, and NodeJS interfaces only. It would be really convenient to save the entire pipeline so that it just takes text as input.

chainyo commented 2 years ago

Hey @alunap thanks for the feedback, much appreciated!

I don't know any way to save a full :hugs: transformers pipeline to ONNX yet. But I have worked a bit with a promising tool that could resolve this use case of packaging everything in one model file. The package is called onnxruntime-extensions and is available here.

With this kind of tool, you can include everything in one onnx model file. It lacks of documentation for the moment, but it's probably a tool that would be auto-included in a near future for people working with ONNX.