search

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
https://inference.readthedocs.io
Apache License 2.0
4.76k stars 373 forks source link

Dash-infer support maybe? #2107

Closed ZB052-A closed 1 week ago

ZB052-A commented 3 weeks ago

Feature request / 功能建议

Is there a subsequent version that can support the Dash-infer inference framework?

Motivation / 动机

I feel that the Dash-infer inference framework can get better performance than llama.cpp inference frameworks, but compatibility can be a bottleneck. https://github.com/modelscope/dash-infer

qinxuye commented 3 weeks ago

PR welcomed for this feature.

github-actions[bot] commented 2 weeks ago

This issue is stale because it has been open for 7 days with no activity.

github-actions[bot] commented 1 week ago

This issue was closed because it has been inactive for 5 days since being marked as stale.