The logit bias option allows you to manually adjust the likelihood of specific tokens appearing in the generated text. By providing a token ID and a positive or negative bias value, you can increase or decrease the probability of that token being generated.
For example, when using the Gemma family of models, use --logit-bias 4521+1 to increase the likelihood of the token 'Hello', or --logit-bias 4521-1 to decrease its likelihood. Using a value of negative infinity, --logit-bias 15043-inf ensures that the token Hello is never produced. A more practical use case might be to prevent the generation of \code{begin} and \code{end} by setting the \ token (235286) to negative infinity with -l 29905-inf.
These usage examples depend on the tokenizer used by the model. The tokenizer-playground space on HF is an easy way to get token ids for commonly used tokenizers.
The logit bias option allows you to manually adjust the likelihood of specific tokens appearing in the generated text. By providing a token ID and a positive or negative bias value, you can increase or decrease the probability of that token being generated.
For example, when using the Gemma family of models, use
--logit-bias 4521+1to increase the likelihood of the token 'Hello', or--logit-bias 4521-1to decrease its likelihood. Using a value of negative infinity,--logit-bias 15043-infensures that the tokenHellois never produced. A more practical use case might be to prevent the generation of\code{begin}and\code{end}by setting the\token (235286) to negative infinity with-l 29905-inf.These usage examples depend on the tokenizer used by the model. The tokenizer-playground space on HF is an easy way to get token ids for commonly used tokenizers.