search

xorbitsai / inference

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
https://inference.readthedocs.io
Apache License 2.0
3.54k stars 294 forks source link

BUG: Fail to run codellama-70b-instruct #996

Open tjlcast opened 5 months ago

tjlcast commented 5 months ago

Describe the bug

When i use xinference to run codellama-70b-instruct. It output a set of unrelated text.

Just like below: image

And there is no anything about codellama-70b-instruct in the doc(https://inference.readthedocs.io/zh-cn/latest/models/builtin/llm/code-llama-instruct.html).

So does Xinference not support codellama-70b-instruct???

If it's due to lack of support, is there any plan to support codellama-70 in the future？

Could you give me a hand about this. Thanks a lot

aresnow1 commented 5 months ago

It's related to chat template, will figure out soon.