search

TensorStack-AI / OnnxStack

C# Stable Diffusion using ONNX Runtime
Apache License 2.0
216 stars 33 forks source link

TextGeneration Pipeline #131

Open saddam213 opened 6 months ago

saddam213 commented 6 months ago

Support for TextGeneration ONNX models

Initial support will be using OnnxRuntime-GenAI onnxruntime-genai

TODO:

AshD commented 6 months ago

Looks cool. How is the text generation speed for onnx models compared to llamasharp for a Mistral 7B v0.2 Instruct model for example?

saddam213 commented 6 months ago

Looks cool. How is the text generation speed for onnx models compared to llamasharp for a Mistral 7B v0.2 Instruct model for example?

Im not too sure, I have not been able to get LLamaSharp working in Web or WPF since version 0.8. so this is my first attempt using another LLM base library, this one is still very new but its a million times less complicated than llama.cpp