Roadmap / Project status / Manifesto / ggml
Inference of Meta's LLaMA model (and others) in pure C/C++
[!IMPORTANT] [2024 Jun 12] Binaries have been renamed w/ a
llama-
prefix.main
is nowllama-cli
,server
isllama-server
, etc (https://github.com/ggerganov/llama.cpp/pull/7809)
llama_token_to_piece
can now optionally render special tokens https://github.com/ggerganov/llama.cpp/pull/6807llama_state_*
https://github.com/ggerganov/llama.cpp/pull/6341llama_synchronize()
+ llama_context_params.n_ubatch
https://github.com/ggerganov/llama.cpp/pull/6017llama_kv_cache_seq_rm()
returns a bool
instead of void
, and new llama_n_seq_max()
returns the upper limit of acceptable seq_id
in batches (relevant when dealing with multiple sequences) https://github.com/ggerganov/llama.cpp/pull/5328struct llama_context_params
https://github.com/ggerganov/llama.cpp/pull/5849convert.py
has been deprecated and moved to examples/convert_legacy_llama.py
, please use convert_hf_to_gguf.py
https://github.com/ggerganov/llama.cpp/pull/7430mmap
support and regenerate imatrix
https://github.com/ggerganov/llama.cpp/pull/6387gguf-split
https://github.com/ggerganov/llama.cpp/discussions/6404The main goal of llama.cpp
is to enable LLM inference with minimal setup and state-of-the-art performance on a wide
variety of hardware - locally and in the cloud.
Since its inception, the project has improved significantly thanks to many contributions. It is the main playground for developing new features for the ggml library.
Supported models:
Typically finetunes of the base models below are supported as well.
(instructions for supporting more models: HOWTO-add-model.md)
Multimodal models:
Bindings:
UI:
Unless otherwise noted these projects are open-source with permissive licensing:
(to have a project listed here, it should clearly state that it depends on llama.cpp
)
Tools:
Infrastructure:
Games:
Here are the end-to-end binary build and model conversion steps for most supported models.
Firstly, you need to get the binary. There are different methods that you can follow:
You can run a basic completion using this command:
llama-cli -m your_model.gguf -p "I believe the meaning of life is" -n 128
# Output:
# I believe the meaning of life is to find your own truth and to live in accordance with it. For me, this means being true to myself and following my passions, even if they don't align with societal expectations. I think that's what I love about yoga – it's not just a physical practice, but a spiritual one too. It's about connecting with yourself, listening to your inner voice, and honoring your own unique journey.
See this page for a full list of parameters.
If you want a more ChatGPT-like experience, you can run in conversation mode by passing -cnv
as a parameter:
llama-cli -m your_model.gguf -p "You are a helpful assistant" -cnv
# Output:
# > hi, who are you?
# Hi there! I'm your helpful assistant! I'm an AI-powered chatbot designed to assist and provide information to users like you. I'm here to help answer your questions, provide guidance, and offer support on a wide range of topics. I'm a friendly and knowledgeable AI, and I'm always happy to help with anything you need. What's on your mind, and how can I assist you today?
#
# > what is 1+1?
# Easy peasy! The answer to 1+1 is... 2!
By default, the chat template will be taken from the input model. If you want to use another chat template, pass --chat-template NAME
as a parameter. See the list of supported templates
./llama-cli -m your_model.gguf -p "You are a helpful assistant" -cnv --chat-template chatml
You can also use your own template via in-prefix, in-suffix and reverse-prompt parameters:
./llama-cli -m your_model.gguf -p "You are a helpful assistant" -cnv --in-prefix 'User: ' --reverse-prompt 'User:'
llama.cpp web server is a lightweight OpenAI API compatible HTTP server that can be used to serve local models and easily connect them to existing clients.
Example usage:
./llama-server -m your_model.gguf --port 8080
# Basic web UI can be accessed via browser: http://localhost:8080
# Chat completion endpoint: http://localhost:8080/v1/chat/completions
[!NOTE] If you prefer basic usage, please consider using conversation mode instead of interactive mode
In this mode, you can always interrupt generation by pressing Ctrl+C and entering one or more lines of text, which will be converted into tokens and appended to the current context. You can also specify a reverse prompt with the parameter -r "reverse prompt string"
. This will result in user input being prompted whenever the exact tokens of the reverse prompt string are encountered in the generation. A typical use is to use a prompt that makes LLaMA emulate a chat between multiple users, say Alice and Bob, and pass -r "Alice:"
.
Here is an example of a few-shot interaction, invoked with the command
# default arguments using a 7B model
./examples/chat.sh
# advanced chat with a 13B model
./examples/chat-13B.sh
# custom arguments using a 13B model
./llama-cli -m ./models/13B/ggml-model-q4_0.gguf -n 256 --repeat_penalty 1.0 --color -i -r "User:" -f prompts/chat-with-bob.txt
Note the use of --color
to distinguish between user input and generated text. Other parameters are explained in more detail in the README for the llama-cli
example program.
The prompt, user inputs, and model generations can be saved and resumed across calls to ./llama-cli
by leveraging --prompt-cache
and --prompt-cache-all
. The ./examples/chat-persistent.sh
script demonstrates this with support for long-running, resumable chat sessions. To use this example, you must provide a file to cache the initial chat prompt and a directory to save the chat session, and may optionally provide the same variables as chat-13B.sh
. The same prompt cache can be reused for new chat sessions. Note that both prompt cache and chat directory are tied to the initial prompt (PROMPT_TEMPLATE
) and the model file.
# Start a new chat
PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default ./examples/chat-persistent.sh
# Resume that chat
PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/default ./examples/chat-persistent.sh
# Start a different chat with the same prompt/model
PROMPT_CACHE_FILE=chat.prompt.bin CHAT_SAVE_DIR=./chat/another ./examples/chat-persistent.sh
# Different prompt cache for different prompt/model
PROMPT_TEMPLATE=./prompts/chat-with-bob.txt PROMPT_CACHE_FILE=bob.prompt.bin \
CHAT_SAVE_DIR=./chat/bob ./examples/chat-persistent.sh
llama.cpp
supports grammars to constrain model output. For example, you can force the model to output JSON only:
./llama-cli -m ./models/13B/ggml-model-q4_0.gguf -n 256 --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'
The grammars/
folder contains a handful of sample grammars. To write your own, check out the GBNF Guide.
For authoring more complex JSON grammars, you can also check out https://grammar.intrinsiclabs.ai/, a browser app that lets you write TypeScript interfaces which it compiles to GBNF grammars that you can save for local use. Note that the app is built and maintained by members of the community, please file any issues or FRs on its repo and not this one.
Please refer to Build llama.cpp locally
Backend | Target devices |
---|---|
Metal | Apple Silicon |
BLAS | All |
BLIS | All |
SYCL | Intel and Nvidia GPU |
MUSA | Moore Threads GPU |
CUDA | Nvidia GPU |
hipBLAS | AMD GPU |
Vulkan | GPU |
CANN | Ascend NPU |
[!NOTE] You can use the GGUF-my-repo space on Hugging Face to quantise your model weights without any setup too. It is synced from
llama.cpp
main every 6 hours.
To obtain the official LLaMA 2 weights please see the Obtaining and using the Facebook LLaMA 2 model section. There is also a large selection of pre-quantized gguf
models available on Hugging Face.
Note: convert.py
has been moved to examples/convert_legacy_llama.py
and shouldn't be used for anything other than Llama/Llama2/Mistral
models and their derivatives.
It does not support LLaMA 3, you can use convert_hf_to_gguf.py
with LLaMA 3 downloaded from Hugging Face.
To learn more about quantizing model, read this documentation
You can use the perplexity
example to measure perplexity over a given prompt (lower perplexity is better).
For more information, see https://huggingface.co/docs/transformers/perplexity.
To learn more how to measure perplexity using llama.cpp, read this documentation
llama.cpp
repo and merge PRs into the master
branchDevelopment documentations
Seminal papers and background on the models
If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT: