Open PaulConyngham opened 1 year ago
The main LLM inference code is in https://github.com/tensorchord/modelz-llm/blob/main/src/modelz_llm/model.py. To add a new model, you need to check https://github.com/tensorchord/llmspec/blob/main/llmspec/model_info.py and add the corresponding docker image in this repo.
as more and more models come out, would be great to see instructions on how we can go about wrapping the model to work with this framework and adding them back