Closed streetycat closed 9 months ago
We can use the follow commands to manage the local compute node with llama:
# add a node /node add llama $model_name $url # remove a node /node rm llama $model_name $url # list all dynamically nodes /node list
Before that , you should download your models, and start the node with follow docker:
docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/llama-model.gguf [ghcr.io/abetlen/llama-cpp-python:latest](http://ghcr.io/abetlen/llama-cpp-python:latest)
For the detail infomation, you can read it at this repository
We can use the follow commands to manage the local compute node with llama:
Before that , you should download your models, and start the node with follow docker:
For the detail infomation, you can read it at this repository