Closed kerthcet closed 3 weeks ago
What would you like to be added:
llama.cpp supports running inference on cpus this is useful for uses has no GPU accelerators, in fact, also helpful to llmaz as we have no GPU servers right now.
What's more, llama.cpp also supports multi-host inference, see https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc
Completion requirements:
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.
/kind feature
/milestone v0.1.0
/assign
Let's support this so we can run e2e tests on cpu machines.
What would you like to be added:
llama.cpp supports running inference on cpus this is useful for uses has no GPU accelerators, in fact, also helpful to llmaz as we have no GPU servers right now.
What's more, llama.cpp also supports multi-host inference, see https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc
Completion requirements:
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.