search

amaiya / onprem

A tool for running on-premises large language models with non-public data
https://amaiya.github.io/onprem
Apache License 2.0
684 stars 32 forks source link

Remove BOS token from default prompt #67

Closed amaiya closed 3 months ago

amaiya commented 3 months ago

The default model (Mistral) specifies <s> as the BOS token in the prompt template: <s>[INST] {prompt} [/INST].

However, this causes llama-cpp-python to output the following warning:

llama_tokenize_internal: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?

... which suggests the BOS token is being added by llama-cpp-python. Thus, <s> should be removed from the default prompt template.