Closed codito closed 1 year ago
Currently there are no plans to add support for llama.cpp and related models. The Basaran project will continue to develop around the Hugging Face ecosystem to be compatible with more existing and future models.
We will also try to introduce more universal technologies to optimize CPU inference.
Thanks @peakji, it makes sense. Closing this issue as out of scope.
Is there a plan to add support for llama.cpp models? It could support inference on CPUs.
https://github.com/ggerganov/llama.cpp and https://github.com/thomasantony/llamacpp-python.
Related: https://github.com/hyperonym/basaran/issues/57