Closed Ben-Epstein closed 4 months ago
Hi @Ben-Epstein, thank you for the feedback! Yes, we do have the RESTful server feature on our roadmap and we are discussing the delivery timeframe for that. In the meantime I am moving this issue into our Discussions forum so that we can continue the discussion
This library is great, I've been testing phi-3-mini-128k, and this by far the fastest runtime for it. For a non-onnx model, id use TGI but presumably you have a more optimized setup for onnx models?
Do you have documentation for best practices around how to deploy the model and handle things like batching, streaming etc? Or, are you planning on building some RESTful server that can be deployed through a docker image?
Might be related to https://github.com/microsoft/onnxruntime-genai/issues/313 but it's not clear what that issue is asking for
Thanks!