tensorchord / modelz-llm

OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)
https://modelz.ai
Apache License 2.0
265 stars 26 forks source link

feat: provide instructions on how community members can wrap models for this project #87

Open PaulConyngham opened 1 year ago

PaulConyngham commented 1 year ago

as more and more models come out, would be great to see instructions on how we can go about wrapping the model to work with this framework and adding them back

kemingy commented 1 year ago

The main LLM inference code is in https://github.com/tensorchord/modelz-llm/blob/main/src/modelz_llm/model.py. To add a new model, you need to check https://github.com/tensorchord/llmspec/blob/main/llmspec/model_info.py and add the corresponding docker image in this repo.