logprobs aren't returned if sampler will reach EOS

[brandon-lockaby]

Prerequisites

llm = Llama(
    model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
    n_gpu_layers=-1,
    seed=8,
    n_ctx=4096,
    logits_all=True,
    kv_overrides={"tokenizer.ggml.eos_token_id": 128002},
)

prompt = """<|start_header_id|>user<|end_header_id|>

What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.

Dogs are typically characterized by their:

1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.

Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:

* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.

Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world

Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""

output = llm(
    prompt,
    echo=False,
    logprobs=100,
    max_tokens=1,

    repeat_penalty=1.0, # disable penalties

    top_k=1,
    #top_p=1.0,
    #min_p=-10000000000.0,
    temperature=0,
)
print(output)

Expected Behavior

logprobs!!

Current Behavior

{'id': 'cmpl-c480f5df-6a62-4a39-9d21-d3058260523b', 'object': 'text_completion', 'created': 1718328460, 'model': '/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf', 'choices': [{'text': '', 'index': 0, 'logprobs': {'tokens': [], 'text_offset': [], 'token_logprobs': [], 'top_logprobs': []}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 430, 'completion_tokens': 0, 'total_tokens': 430}}

Environment and Context

llama_cpp_python-0.2.78

CUDA 12.4

[brandon-lockaby]

I tried to use kv_overrides to change the EOS token, here, as there doesn't seem to be any other way, but that didn't do the trick

[brandon-lockaby]

from llama_cpp import Llama

llm = Llama(
    model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
    n_gpu_layers=-1,
    seed=8,
    n_ctx=4096,
    logits_all=True,
 )

prompt = """<|start_header_id|>user<|end_header_id|>

What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.

Dogs are typically characterized by their:

1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.

Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:

* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.

Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world

Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""

output = llm(
    prompt,
    echo=False,
    logprobs=100,
    max_tokens=1,

    repeat_penalty=1.0, # disable penalties

    top_k=1,
    temperature=0,
)
print(output)

[brandon-lockaby]

topk=1 or temperature=0 or other variations of these parameters don't make a difference, EOS causes generation to come back blank with no logprobs

[brandon-lockaby]

Exacerbatingly, trying to change EOS doesn't work:

llm = Llama(
    model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
    n_gpu_layers=-1,
    seed=8,
    n_ctx=4096,
    logits_all=True,
    kv_overrides={"tokenizer.ggml.eos_token_id": 0},
 )

[brandon-lockaby]

output = llm(
    prompt,
    echo=False,
    logprobs=100,
    max_tokens=1,

   repeat_penalty=1.0, # disable penalties
   temperature=0,

   logit_bias={128001: -1000}
)

Logit bias down the EOS id doesn't work either, idk 🤷

[LeoPerelli]

Same issue, no logprobs returned. I was getting an error when using the call method of LLama + logits_all = True. Had to switch to the create_chat_completion method, no more error but still no logprobs.