search

iaalm / llama-api-server

A OpenAI API compatible REST server for llama.
MIT License
199 stars 12 forks source link

[FEATURE] Upgrade llama-cpp-python to latest version #49

Open RomainMendez opened 1 year ago

RomainMendez commented 1 year ago

The version of llama-cpp-python this project uses is quite old. Therefore I get a lot of errors regarding versions of GGML models. It also doesn't support GGUF models.

I would suggest to up the version of llama-cpp-python to the latest one.

GGUF models are the future anyway it would seem. Perhaps we could work on something that allows chosing ?

iaalm commented 1 year ago

Thanks for remaindering.👍

iaalm commented 1 year ago

This project actually depends on llamacpp-python. Seems their most recent version doesn't support gguf. I will see whether they can have an update.