feat: [support log prob like OpenAI API]

nguyenhoangthuan99 commented 3 weeks ago

Problem The api need to support logprobs and top_logprobs https://platform.openai.com/docs/api-reference/chat/create#chat-create-logprobs

The response object for non stream mode: https://platform.openai.com/docs/api-reference/chat/object#chat/object-choices
The response object for stream mode: https://platform.openai.com/docs/api-reference/chat/streaming#chat/streaming-choices related issue https://github.com/janhq/internal/issues/160

gabrielle-ong commented 1 week ago

Hi @nguyenhoangthuan99, I tested the output of Logprobs for OpenAI vs Cortex. Cortex: tried with different models - llama3.2-3b, mistral7b, tinyllama Problem: Logprobs always returns as 0.0 or 1.0 Expected behavior (OpenAI) - Logprobs should be negative numbers.

Output:

Expected (OpenAI):

Cortex:

Code:

## from https://cookbook.openai.com/examples/using_logprobs
from openai import OpenAI
from math import exp
import os
import numpy as np

## OpenAI
# MODEL = "gpt-4o" 
# client = OpenAI(
#     api_key="xxx"
# )

## Cortex
ENDPOINT = "http://localhost:39281/v1"
MODEL = "llama3.2:3b-gguf-q2-k"

client = OpenAI(
    base_url=ENDPOINT,
    api_key="not-needed"
)

def get_completion(
    messages: list[dict[str, str]],
    model: str = MODEL,
    max_tokens=500,
    temperature=0,
    stop=None,
    seed=123,
    tools=None,
    logprobs=None,  # whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message..
    top_logprobs=None,
) -> str:
    params = {
        "messages": messages,
        "model": model,  # Use the parameter 'model' instead of the global 'MODEL'
        "max_tokens": max_tokens,
        "temperature": temperature,
        "stop": stop,
        "seed": seed,
        "logprobs": logprobs,
        "top_logprobs": top_logprobs,
    }
    if tools:
        params["tools"] = tools

    completion = client.chat.completions.create(**params)
    return completion

CLASSIFICATION_PROMPT = """You will be given a headline of a news article.
Classify the article into one of the following categories: Technology, Politics, Sports, and Art.
Return only the name of the category, and nothing else.
MAKE SURE your output is one of the four categories stated.
Article headline: {headline}"""

headlines = [
    "Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.",
    "Local Mayor Launches Initiative to Enhance Urban Public Transport.",
    "Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut",
    "Dog eats Cat in a fight",
]

for headline in headlines:
    print(f"\nHeadline: {headline}")
    API_RESPONSE = get_completion(
        [{"role": "user", "content": CLASSIFICATION_PROMPT.format(headline=headline)}],
        model=MODEL,
        logprobs=True,
        top_logprobs=3,
    )
    top_two_logprobs = API_RESPONSE.choices[0].logprobs.content[0].top_logprobs
    html_content = ""
    for i, logprob in enumerate(top_two_logprobs, start=1):
        print(
            f"Output token {i}: {logprob.token}, "
            f"logprobs: {logprob.logprob}, "
            f"linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%"
        )

gabrielle-ong commented 1 week ago

Also tested with the code snippet in your PR: https://github.com/janhq/cortex.llamacpp/pull/276,

OpenAI returns negative logprobs while Cortex returns positive (0.2, 0.3 etc which rounds to 0.0)

Cortex response:

hatCompletion(id='f3z6iZdxgXSFKI0YS9Wx', choices=[Choice(finish_reason='stop', index=0, logprobs=ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=0.9762434363365173, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=0.9762434363365173), TopLogprob(token='A', bytes=[65], logprob=0.02375652827322483)]), ChatCompletionTokenLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.816756010055542, top_logprobs=[TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.816756010055542), TopLogprob(token=' ', bytes=[32], logprob=0.18324396014213562)]), ChatCompletionTokenLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=1.0), TopLogprob(token=' Angels', bytes=[32, 65, 110, 103, 101, 108, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0), TopLogprob(token=' Lakers', bytes=[32, 76, 97, 107, 101, 114, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' did', bytes=[32, 100, 105, 100], logprob=0.5692539215087891, top_logprobs=[TopLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.5692539215087891), TopLogprob(token=' did', bytes=[32, 100, 105, 100], logprob=0.37212011218070984)]), ChatCompletionTokenLogprob(token=' not', bytes=[32, 110, 111, 116], logprob=1.0, top_logprobs=[TopLogprob(token=' not', bytes=[32, 110, 111, 116], logprob=1.0), TopLogprob(token=' win', bytes=[32, 119, 105, 110], logprob=0.0)]), ChatCompletionTokenLogprob(token=' win', bytes=[32, 119, 105, 110], logprob=1.0, top_logprobs=[TopLogprob(token=' win', bytes=[32, 119, 105, 110], logprob=1.0), TopLogprob(token=' appear', bytes=[32, 97, 112, 112, 101, 97, 114], logprob=0.0)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.950188159942627, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.950188159942627), TopLogprob(token=' a', bytes=[32, 97], logprob=0.03197034075856209)]), ChatCompletionTokenLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=1.0, top_logprobs=[TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=1.0), TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=1.0), TopLogprob(token="'s", bytes=[39, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=0.9248781800270081, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.9248781800270081), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.037800271064043045)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=1.0, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=1.0), TopLogprob(token=' a', bytes=[32, 97], logprob=0.0)]), ChatCompletionTokenLogprob(token='202', bytes=[50, 48, 50], logprob=1.0, top_logprobs=[TopLogprob(token='202', bytes=[50, 48, 50], logprob=1.0), TopLogprob(token='201', bytes=[50, 48, 49], logprob=0.0)]), ChatCompletionTokenLogprob(token='0', bytes=[48], logprob=1.0, top_logprobs=[TopLogprob(token='0', bytes=[48], logprob=1.0), TopLogprob(token='1', bytes=[49], logprob=0.0)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=0.6126563549041748, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=0.6126563549041748), TopLogprob(token=',', bytes=[44], logprob=0.3426532447338104)]), ChatCompletionTokenLogprob(token=' The', bytes=[32, 84, 104, 101], logprob=0.9089756608009338, top_logprobs=[TopLogprob(token=' The', bytes=[32, 84, 104, 101], logprob=0.9089756608009338), TopLogprob(token=' However', bytes=[32, 72, 111, 119, 101, 118, 101, 114], logprob=0.07462749630212784)]), ChatCompletionTokenLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=1.0), TopLogprob(token=' team', bytes=[32, 116, 101, 97, 109], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=1.0), TopLogprob(token=' Angels', bytes=[32, 65, 110, 103, 101, 108, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0), TopLogprob(token=' Lakers', bytes=[32, 76, 97, 107, 101, 114, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=0.3067297041416168, top_logprobs=[TopLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=0.3067297041416168), TopLogprob(token=' lost', bytes=[32, 108, 111, 115, 116], logprob=0.2009105533361435)]), ChatCompletionTokenLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.443875789642334, top_logprobs=[TopLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.443875789642334), TopLogprob(token=' played', bytes=[32, 112, 108, 97, 121, 101, 100], logprob=0.21468976140022278)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9565920233726501, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9565920233726501), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=0.03001466579735279)]), ChatCompletionTokenLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.9042747616767883, top_logprobs=[TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.9042747616767883), TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.09572524577379227)]), ChatCompletionTokenLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=1.0), TopLogprob(token=' series', bytes=[32, 115, 101, 114, 105, 101, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=0.5215603113174438, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.5215603113174438), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.4470324218273163)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=1.0, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=1.0), TopLogprob(token=' a', bytes=[32, 97], logprob=0.0)]), ChatCompletionTokenLogprob(token='202', bytes=[50, 48, 50], logprob=1.0, top_logprobs=[TopLogprob(token='202', bytes=[50, 48, 50], logprob=1.0), TopLogprob(token='201', bytes=[50, 48, 49], logprob=0.0)]), ChatCompletionTokenLogprob(token='0', bytes=[48], logprob=0.956136167049408, top_logprobs=[TopLogprob(token='0', bytes=[48], logprob=0.956136167049408), TopLogprob(token='2', bytes=[50], logprob=0.04386384040117264)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.929105281829834, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.929105281829834), TopLogprob(token=' by', bytes=[32, 98, 121], logprob=0.06212448328733444)]), ChatCompletionTokenLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=0.6098390817642212, top_logprobs=[TopLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=0.6098390817642212), TopLogprob(token=' as', bytes=[32, 97, 115], logprob=0.14639152586460114)]), ChatCompletionTokenLogprob(token=' I', bytes=[32, 73], logprob=0.3974471986293793, top_logprobs=[TopLogprob(token=' I', bytes=[32, 73], logprob=0.3974471986293793), TopLogprob(token=' it', bytes=[32, 105, 116], logprob=0.3459700047969818)]), ChatCompletionTokenLogprob(token="'ll", bytes=[39, 108, 108], logprob=0.7927050590515137, top_logprobs=[TopLogprob(token="'ll", bytes=[39, 108, 108], logprob=0.7927050590515137), TopLogprob(token="'m", bytes=[39, 109], logprob=0.12399910390377045)]), ChatCompletionTokenLogprob(token=' explain', bytes=[32, 101, 120, 112, 108, 97, 105, 110], logprob=0.9603455662727356, top_logprobs=[TopLogprob(token=' explain', bytes=[32, 101, 120, 112, 108, 97, 105, 110], logprob=0.9603455662727356), TopLogprob(token=' get', bytes=[32, 103, 101, 116], logprob=0.03965437784790993)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.8817760348320007, top_logprobs=[TopLogprob(token=' why', bytes=[32, 119, 104, 121], logprob=0.8817760348320007), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.07975482940673828)]), ChatCompletionTokenLogprob(token=' later', bytes=[32, 108, 97, 116, 101, 114], logprob=0.41886526346206665, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.41886526346206665), TopLogprob(token=' later', bytes=[32, 108, 97, 116, 101, 114], logprob=0.3193166255950928)]), ChatCompletionTokenLogprob(token='.\n\n', bytes=[46, 10, 10], logprob=0.9911603331565857, top_logprobs=[TopLogprob(token='.\n\n', bytes=[46, 10, 10], logprob=0.9911603331565857), TopLogprob(token='.', bytes=[46], logprob=0.008839697577059269)]), ChatCompletionTokenLogprob(token='However', bytes=[72, 111, 119, 101, 118, 101, 114], logprob=0.5082773566246033, top_logprobs=[TopLogprob(token='However', bytes=[72, 111, 119, 101, 118, 101, 114], logprob=0.5082773566246033), TopLogprob(token='The', bytes=[84, 104, 101], logprob=0.21322138607501984)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=1.0, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=1.0), TopLogprob(token=' it', bytes=[32, 105, 116], logprob=0.0)]), ChatCompletionTokenLogprob(token=' if', bytes=[32, 105, 102], logprob=0.3823017179965973, top_logprobs=[TopLogprob(token=' it', bytes=[32, 105, 116], logprob=0.3823017179965973), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.3128318190574646)]), ChatCompletionTokenLogprob(token=' you', bytes=[32, 121, 111, 117], logprob=0.8714560270309448, top_logprobs=[TopLogprob(token=' you', bytes=[32, 121, 111, 117], logprob=0.8714560270309448), TopLogprob(token=' we', bytes=[32, 119, 101], logprob=0.1285439431667328)]), ChatCompletionTokenLogprob(token="'re", bytes=[39, 114, 101], logprob=0.9579666256904602, top_logprobs=[TopLogprob(token="'re", bytes=[39, 114, 101], logprob=0.9579666256904602), TopLogprob(token=' are', bytes=[32, 97, 114, 101], logprob=0.02733428403735161)]), ChatCompletionTokenLogprob(token=' referring', bytes=[32, 114, 101, 102, 101, 114, 114, 105, 110, 103], logprob=0.5727916955947876, top_logprobs=[TopLogprob(token=' referring', bytes=[32, 114, 101, 102, 101, 114, 114, 105, 110, 103], logprob=0.5727916955947876), TopLogprob(token=' asking', bytes=[32, 97, 115, 107, 105, 110, 103], logprob=0.31906208395957947)]), ChatCompletionTokenLogprob(token=' to', bytes=[32, 116, 111], logprob=1.0, top_logprobs=[TopLogprob(token=' to', bytes=[32, 116, 111], logprob=1.0), TopLogprob(token=' specifically', bytes=[32, 115, 112, 101, 99, 105, 102, 105, 99, 97, 108, 108, 121], logprob=0.0)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=0.6905534267425537, top_logprobs=[TopLogprob(token=' a', bytes=[32, 97], logprob=0.6905534267425537), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.30944663286209106)]), ChatCompletionTokenLogprob(token=' non', bytes=[32, 110, 111, 110], logprob=0.9133797883987427, top_logprobs=[TopLogprob(token=' different', bytes=[32, 100, 105, 102, 102, 101, 114, 101, 110, 116], logprob=0.9133797883987427), TopLogprob(token=' specific', bytes=[32, 115, 112, 101, 99, 105, 102, 105, 99], logprob=0.024693142622709274)]), ChatCompletionTokenLogprob(token='-US', bytes=[45, 85, 83], logprob=0.3617076873779297, top_logprobs=[TopLogprob(token='-', bytes=[45], logprob=0.3617076873779297), TopLogprob(token='-tr', bytes=[45, 116, 114], logprob=0.20019105076789856)]), ChatCompletionTokenLogprob(token='-based', bytes=[45, 98, 97, 115, 101, 100], logprob=0.5976563692092896, top_logprobs=[TopLogprob(token='-based', bytes=[45, 98, 97, 115, 101, 100], logprob=0.5976563692092896), TopLogprob(token=' based', bytes=[32, 98, 97, 115, 101, 100], logprob=0.22212395071983337)]), ChatCompletionTokenLogprob(token=' team', bytes=[32, 116, 101, 97, 109], logprob=1.0, top_logprobs=[TopLogprob(token=' team', bytes=[32, 116, 101, 97, 109], logprob=1.0), TopLogprob(token=' organization', bytes=[32, 111, 114, 103, 97, 110, 105, 122, 97, 116, 105, 111, 110], logprob=0.0)]), ChatCompletionTokenLogprob(token=' winning', bytes=[32, 119, 105, 110, 110, 105, 110, 103], logprob=0.9087316989898682, top_logprobs=[TopLogprob(token=' winning', bytes=[32, 119, 105, 110, 110, 105, 110, 103], logprob=0.9087316989898682), TopLogprob(token=',', bytes=[44], logprob=0.07384128868579865)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9339510202407837, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9339510202407837), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=0.03833436965942383)]), ChatCompletionTokenLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.7401295304298401, top_logprobs=[TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.7401295304298401), TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.22117087244987488)]), ChatCompletionTokenLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=0.9803146123886108, top_logprobs=[TopLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=0.9803146123886108), TopLogprob(token="'s", bytes=[39, 115], logprob=0.01968538388609886)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.49547016620635986, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.49547016620635986), TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.393582284450531)]), ChatCompletionTokenLogprob(token=' then', bytes=[32, 116, 104, 101, 110], logprob=0.33730170130729675, top_logprobs=[TopLogprob(token=' I', bytes=[32, 73], logprob=0.33730170130729675), TopLogprob(token=' it', bytes=[32, 105, 116], logprob=0.28797832131385803)]), ChatCompletionTokenLogprob(token=' it', bytes=[32, 105, 116], logprob=0.35244351625442505, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.35244351625442505), TopLogprob(token=' it', bytes=[32, 105, 116], logprob=0.3093152344226837)]), ChatCompletionTokenLogprob(token=' was', bytes=[32, 119, 97, 115], logprob=0.6463494896888733, top_logprobs=[TopLogprob(token=' was', bytes=[32, 119, 97, 115], logprob=0.6463494896888733), TopLogprob(token="'s", bytes=[39, 115], logprob=0.3536505103111267)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9007476568222046, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9007476568222046), TopLogprob(token=' a', bytes=[32, 97], logprob=0.0593450628221035)]), ChatCompletionTokenLogprob(token=' **', bytes=[32, 42, 42], logprob=0.4961835443973541, top_logprobs=[TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.4961835443973541), TopLogprob(token=' first', bytes=[32, 102, 105, 114, 115, 116], logprob=0.3262627422809601)]), ChatCompletionTokenLogprob(token='Los', bytes=[76, 111, 115], logprob=0.9836245775222778, top_logprobs=[TopLogprob(token='Los', bytes=[76, 111, 115], logprob=0.9836245775222778), TopLogprob(token='L', bytes=[76], logprob=0.01637542061507702)]), ChatCompletionTokenLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=1.0), TopLogprob(token=' Angels', bytes=[32, 65, 110, 103, 101, 108, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0), TopLogprob(token=' Lakers', bytes=[32, 76, 97, 107, 101, 114, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token='**', bytes=[42, 42], logprob=0.7888962030410767, top_logprobs=[TopLogprob(token='**', bytes=[42, 42], logprob=0.7888962030410767), TopLogprob(token=' who', bytes=[32, 119, 104, 111], logprob=0.09024917334318161)]), ChatCompletionTokenLogprob(token=' who', bytes=[32, 119, 104, 111], logprob=0.9773011803627014, top_logprobs=[TopLogprob(token=' who', bytes=[32, 119, 104, 111], logprob=0.9773011803627014), TopLogprob(token=' (', bytes=[32, 40], logprob=0.0125479931011796)]), ChatCompletionTokenLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.9430122375488281, top_logprobs=[TopLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.9430122375488281), TopLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=0.023815300315618515)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9041731357574463, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9041731357574463), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=0.06639248877763748)]), ChatCompletionTokenLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.5952978134155273, top_logprobs=[TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.5952978134155273), TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.36497706174850464)]), ChatCompletionTokenLogprob(token='!\n\n', bytes=[33, 10, 10], logprob=0.3287578225135803, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.3287578225135803), TopLogprob(token='!\n\n', bytes=[33, 10, 10], logprob=0.21022166311740875)]), ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=0.8573283553123474, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=0.8573283553123474), TopLogprob(token='In', bytes=[73, 110], logprob=0.04830409586429596)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.8761915564537048, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.8761915564537048), TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.06210128217935562)]), ChatCompletionTokenLogprob(token=' defeated', bytes=[32, 100, 101, 102, 101, 97, 116, 101, 100], logprob=0.739148736000061, top_logprobs=[TopLogprob(token=' defeated', bytes=[32, 100, 101, 102, 101, 97, 116, 101, 100], logprob=0.739148736000061), TopLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.07670271396636963)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=1.0, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=1.0), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=0.0)]), ChatCompletionTokenLogprob(token=' **', bytes=[32, 42, 42], logprob=0.47027382254600525, top_logprobs=[TopLogprob(token=' **', bytes=[32, 42, 42], logprob=0.47027382254600525), TopLogprob(token=' Tampa', bytes=[32, 84, 97, 109, 112, 97], logprob=0.46046772599220276)]), ChatCompletionTokenLogprob(token='San', bytes=[83, 97, 110], logprob=0.2905765175819397, top_logprobs=[TopLogprob(token='New', bytes=[78, 101, 119], logprob=0.2905765175819397), TopLogprob(token='Boston', bytes=[66, 111, 115, 116, 111, 110], logprob=0.25053659081459045)]), ChatCompletionTokenLogprob(token=' Diego', bytes=[32, 68, 105, 101, 103, 111], logprob=1.0, top_logprobs=[TopLogprob(token=' Diego', bytes=[32, 68, 105, 101, 103, 111], logprob=1.0), TopLogprob(token=' Francisco', bytes=[32, 70, 114, 97, 110, 99, 105, 115, 99, 111], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Padres', bytes=[32, 80, 97, 100, 114, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Padres', bytes=[32, 80, 97, 100, 114, 101, 115], logprob=1.0), TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token='**,', bytes=[42, 42, 44], logprob=0.7513446807861328, top_logprobs=[TopLogprob(token='**,', bytes=[42, 42, 44], logprob=0.7513446807861328), TopLogprob(token='**', bytes=[42, 42], logprob=0.2486553192138672)]), ChatCompletionTokenLogprob(token=' and', bytes=[32, 97, 110, 100], logprob=0.24803514778614044, top_logprobs=[TopLogprob(token=' not', bytes=[32, 110, 111, 116], logprob=0.24803514778614044), TopLogprob(token=' ', bytes=[32], logprob=0.24745723605155945)]), ChatCompletionTokenLogprob(token=' more', bytes=[32, 109, 111, 114, 101], logprob=0.23848354816436768, top_logprobs=[TopLogprob(token=' more', bytes=[32, 109, 111, 114, 101], logprob=0.23848354816436768), TopLogprob(token=' then', bytes=[32, 116, 104, 101, 110], logprob=0.2189086675643921)]), ChatCompletionTokenLogprob(token=' specifically', bytes=[32, 115, 112, 101, 99, 105, 102, 105, 99, 97, 108, 108, 121], logprob=0.9628136157989502, top_logprobs=[TopLogprob(token=' specifically', bytes=[32, 115, 112, 101, 99, 105, 102, 105, 99, 97, 108, 108, 121], logprob=0.9628136157989502), TopLogprob(token=' importantly', bytes=[32, 105, 109, 112, 111, 114, 116, 97, 110, 116, 108, 121], logprob=0.03718643635511398)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.9893802404403687, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.9893802404403687), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.010619714856147766)]), ChatCompletionTokenLogprob(token=' they', bytes=[32, 116, 104, 101, 121], logprob=0.7805576920509338, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.7805576920509338), TopLogprob(token=' they', bytes=[32, 116, 104, 101, 121], logprob=0.13257759809494019)]), ChatCompletionTokenLogprob(token=' beat', bytes=[32, 98, 101, 97, 116], logprob=0.6579117178916931, top_logprobs=[TopLogprob(token=' defeated', bytes=[32, 100, 101, 102, 101, 97, 116, 101, 100], logprob=0.6579117178916931), TopLogprob(token=' beat', bytes=[32, 98, 101, 97, 116], logprob=0.10870964080095291)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9103614091873169, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.9103614091873169), TopLogprob(token=' them', bytes=[32, 116, 104, 101, 109], logprob=0.042625706642866135)]), ChatCompletionTokenLogprob(token=' San', bytes=[32, 83, 97, 110], logprob=0.38623857498168945, top_logprobs=[TopLogprob(token=' Padres', bytes=[32, 80, 97, 100, 114, 101, 115], logprob=0.38623857498168945), TopLogprob(token=' **', bytes=[32, 42, 42], logprob=0.35078322887420654)]), ChatCompletionTokenLogprob(token=' Diego', bytes=[32, 68, 105, 101, 103, 111], logprob=1.0, top_logprobs=[TopLogprob(token=' Diego', bytes=[32, 68, 105, 101, 103, 111], logprob=1.0), TopLogprob(token=' Francisco', bytes=[32, 70, 114, 97, 110, 99, 105, 115, 99, 111], logprob=0.0)]), ChatCompletionTokenLogprob(token=' Padres', bytes=[32, 80, 97, 100, 114, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Padres', bytes=[32, 80, 97, 100, 114, 101, 115], logprob=1.0), TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token="'", bytes=[39], logprob=0.8195392489433289, top_logprobs=[TopLogprob(token="'", bytes=[39], logprob=0.8195392489433289), TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.08358334749937057)]), ChatCompletionTokenLogprob(token=' counterpart', bytes=[32, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116], logprob=0.2098115086555481, top_logprobs=[TopLogprob(token=' manager', bytes=[32, 109, 97, 110, 97, 103, 101, 114], logprob=0.2098115086555481), TopLogprob(token=' ace', bytes=[32, 97, 99, 101], logprob=0.10836197435855865)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.6728065609931946, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.6728065609931946), TopLogprob(token=' (', bytes=[32, 40], logprob=0.0850725844502449)]), ChatCompletionTokenLogprob(token=' **', bytes=[32, 42, 42], logprob=0.8922825455665588, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.8922825455665588), TopLogprob(token=' **', bytes=[32, 42, 42], logprob=0.09291849285364151)]), ChatCompletionTokenLogprob(token='the', bytes=[116, 104, 101], logprob=0.9024651050567627, top_logprobs=[TopLogprob(token='the', bytes=[116, 104, 101], logprob=0.9024651050567627), TopLogprob(token='T', bytes=[84], logprob=0.023337168619036674)]), ChatCompletionTokenLogprob(token=' Arizona', bytes=[32, 65, 114, 105, 122, 111, 110, 97], logprob=0.265689879655838, top_logprobs=[TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.265689879655838), TopLogprob(token=' Tampa', bytes=[32, 84, 97, 109, 112, 97], logprob=0.2442207783460617)]), ChatCompletionTokenLogprob(token=' Diamond', bytes=[32, 68, 105, 97, 109, 111, 110, 100], logprob=1.0, top_logprobs=[TopLogprob(token=' Diamond', bytes=[32, 68, 105, 97, 109, 111, 110, 100], logprob=1.0), TopLogprob(token=' D', bytes=[32, 68], logprob=0.0)]), ChatCompletionTokenLogprob(token='backs', bytes=[98, 97, 99, 107, 115], logprob=1.0, top_logprobs=[TopLogprob(token='backs', bytes=[98, 97, 99, 107, 115], logprob=1.0), TopLogprob(token='back', bytes=[98, 97, 99, 107], logprob=0.0)]), ChatCompletionTokenLogprob(token='**,', bytes=[42, 42, 44], logprob=0.43109142780303955, top_logprobs=[TopLogprob(token='**', bytes=[42, 42], logprob=0.43109142780303955), TopLogprob(token='**,', bytes=[42, 42, 44], logprob=0.2677830755710602)]), ChatCompletionTokenLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=0.853933572769165, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.853933572769165), TopLogprob(token=' but', bytes=[32, 98, 117, 116], logprob=0.04290930926799774)]), ChatCompletionTokenLogprob(token=' not', bytes=[32, 110, 111, 116], logprob=0.25855210423469543, top_logprobs=[TopLogprob(token=' ultimately', bytes=[32, 117, 108, 116, 105, 109, 97, 116, 101, 108, 121], logprob=0.25855210423469543), TopLogprob(token=' most', bytes=[32, 109, 111, 115, 116], logprob=0.24820931255817413)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=0.5880088210105896, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.5880088210105896), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=0.12575024366378784)]), ChatCompletionTokenLogprob(token=' a', bytes=[32, 97], logprob=0.6535693407058716, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.6535693407058716), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.15835878252983093)]), ChatCompletionTokenLogprob(token=' game', bytes=[32, 103, 97, 109, 101], logprob=0.2471202313899994, top_logprobs=[TopLogprob(token=' game', bytes=[32, 103, 97, 109, 101], logprob=0.2471202313899994), TopLogprob(token=' traditional', bytes=[32, 116, 114, 97, 100, 105, 116, 105, 111, 110, 97, 108], logprob=0.1570872664451599)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=0.42304933071136475, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=0.42304933071136475), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.129036545753479)]), ChatCompletionTokenLogprob(token=' The', bytes=[32, 84, 104, 101], logprob=0.7205371260643005, top_logprobs=[TopLogprob(token=' The', bytes=[32, 84, 104, 101], logprob=0.7205371260643005), TopLogprob(token=' \n\n', bytes=[32, 10, 10], logprob=0.07354351133108139)]), ChatCompletionTokenLogprob(token=' winner', bytes=[32, 119, 105, 110, 110, 101, 114], logprob=0.7341276407241821, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.7341276407241821), TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.06161559000611305)]), ChatCompletionTokenLogprob(token=' of', bytes=[32, 111, 102], logprob=0.9669046998023987, top_logprobs=[TopLogprob(token=' of', bytes=[32, 111, 102], logprob=0.9669046998023987), TopLogprob(token=' was', bytes=[32, 119, 97, 115], logprob=0.02637360990047455)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.5512768626213074, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.5512768626213074), TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.32390469312667847)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=0.7899599671363831, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=0.7899599671363831), TopLogprob(token=' series', bytes=[32, 115, 101, 114, 105, 101, 115], logprob=0.12961064279079437)]), ChatCompletionTokenLogprob(token='202', bytes=[50, 48, 50], logprob=1.0, top_logprobs=[TopLogprob(token='202', bytes=[50, 48, 50], logprob=1.0), TopLogprob(token='201', bytes=[50, 48, 49], logprob=0.0)]), ChatCompletionTokenLogprob(token='0', bytes=[48], logprob=1.0, top_logprobs=[TopLogprob(token='0', bytes=[48], logprob=1.0), TopLogprob(token='1', bytes=[49], logprob=0.0)]), ChatCompletionTokenLogprob(token=' world', bytes=[32, 119, 111, 114, 108, 100], logprob=0.590471088886261, top_logprobs=[TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.590471088886261), TopLogprob(token=' season', bytes=[32, 115, 101, 97, 115, 111, 110], logprob=0.335599422454834)]), ChatCompletionTokenLogprob(token=' series', bytes=[32, 115, 101, 114, 105, 101, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' series', bytes=[32, 115, 101, 114, 105, 101, 115], logprob=1.0), TopLogprob(token="'s", bytes=[39, 115], logprob=0.0)]), ChatCompletionTokenLogprob(token=' was', bytes=[32, 119, 97, 115], logprob=0.5576503872871399, top_logprobs=[TopLogprob(token=' is', bytes=[32, 105, 115], logprob=0.5576503872871399), TopLogprob(token=' was', bytes=[32, 119, 97, 115], logprob=0.21701419353485107)]), ChatCompletionTokenLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=0.6842241287231445, top_logprobs=[TopLogprob(token=' actually', bytes=[32, 97, 99, 116, 117, 97, 108, 108, 121], logprob=0.6842241287231445), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.17528308928012848)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.7532948851585388, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.7532948851585388), TopLogprob(token='...', bytes=[46, 46, 46], logprob=0.09030400216579437)]), ChatCompletionTokenLogprob(token=' **', bytes=[32, 42, 42], logprob=0.7583876252174377, top_logprobs=[TopLogprob(token=' **', bytes=[32, 42, 42], logprob=0.7583876252174377), TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=0.16296911239624023)]), ChatCompletionTokenLogprob(token='Los', bytes=[76, 111, 115], logprob=0.7245551943778992, top_logprobs=[TopLogprob(token='Los', bytes=[76, 111, 115], logprob=0.7245551943778992), TopLogprob(token='L', bytes=[76], logprob=0.07419734448194504)]), ChatCompletionTokenLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=0.984880805015564, top_logprobs=[TopLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=0.984880805015564), TopLogprob(token=' AN', bytes=[32, 65, 78], logprob=0.007980608381330967)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=1.0), TopLogprob(token=' D', bytes=[32, 68], logprob=0.0)]), ChatCompletionTokenLogprob(token='**', bytes=[42, 42], logprob=0.4997205436229706, top_logprobs=[TopLogprob(token='**', bytes=[42, 42], logprob=0.4997205436229706), TopLogprob(token='**,', bytes=[42, 42, 44], logprob=0.4121934175491333)]), ChatCompletionTokenLogprob(token='.\n\n', bytes=[46, 10, 10], logprob=0.798690140247345, top_logprobs=[TopLogprob(token='.\n\n', bytes=[46, 10, 10], logprob=0.798690140247345), TopLogprob(token='.', bytes=[46], logprob=0.04479764774441719)]), ChatCompletionTokenLogprob(token='However', bytes=[72, 111, 119, 101, 118, 101, 114], logprob=0.4527271091938019, top_logprobs=[TopLogprob(token='However', bytes=[72, 111, 119, 101, 118, 101, 114], logprob=0.4527271091938019), TopLogprob(token='In', bytes=[73, 110], logprob=0.10240784287452698)]), ChatCompletionTokenLogprob(token=',', bytes=[44], logprob=0.9478417634963989, top_logprobs=[TopLogprob(token=',', bytes=[44], logprob=0.9478417634963989), TopLogprob(token=' I', bytes=[32, 73], logprob=0.03249528259038925)]), ChatCompletionTokenLogprob(token=' I', bytes=[32, 73], logprob=0.46125999093055725, top_logprobs=[TopLogprob(token=' I', bytes=[32, 73], logprob=0.46125999093055725), TopLogprob(token=' if', bytes=[32, 105, 102], logprob=0.32872214913368225)]), ChatCompletionTokenLogprob(token=' will', bytes=[32, 119, 105, 108, 108], logprob=0.22300826013088226, top_logprobs=[TopLogprob(token=' think', bytes=[32, 116, 104, 105, 110, 107], logprob=0.22300826013088226), TopLogprob(token=' must', bytes=[32, 109, 117, 115, 116], logprob=0.21682707965373993)]), ChatCompletionTokenLogprob(token=' explain', bytes=[32, 101, 120, 112, 108, 97, 105, 110], logprob=0.6963526606559753, top_logprobs=[TopLogprob(token=' explain', bytes=[32, 101, 120, 112, 108, 97, 105, 110], logprob=0.6963526606559753), TopLogprob(token=' correct', bytes=[32, 99, 111, 114, 114, 101, 99, 116], logprob=0.10305464267730713)]), ChatCompletionTokenLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.45814576745033264, top_logprobs=[TopLogprob(token=' that', bytes=[32, 116, 104, 97, 116], logprob=0.45814576745033264), TopLogprob(token=' why', bytes=[32, 119, 104, 121], logprob=0.1635449081659317)]), ChatCompletionTokenLogprob(token=' it', bytes=[32, 105, 116], logprob=0.18574489653110504, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.18574489653110504), TopLogprob(token=' later', bytes=[32, 108, 97, 116, 101, 114], logprob=0.16818848252296448)]), ChatCompletionTokenLogprob(token="'s", bytes=[39, 115], logprob=0.33917248249053955, top_logprobs=[TopLogprob(token=' wasn', bytes=[32, 119, 97, 115, 110], logprob=0.33917248249053955), TopLogprob(token="'s", bytes=[39, 115], logprob=0.25537022948265076)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.1976018100976944, top_logprobs=[TopLogprob(token=' not', bytes=[32, 110, 111, 116], logprob=0.1976018100976944), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.13976113498210907)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.20133285224437714, top_logprobs=[TopLogprob(token=' correct', bytes=[32, 99, 111, 114, 114, 101, 99, 116], logprob=0.20133285224437714), TopLogprob(token=' same', bytes=[32, 115, 97, 109, 101], logprob=0.14385125041007996)]), ChatCompletionTokenLogprob(token=' who', bytes=[32, 119, 104, 111], logprob=0.7399000525474548, top_logprobs=[TopLogprob(token=' who', bytes=[32, 119, 104, 111], logprob=0.7399000525474548), TopLogprob(token="'", bytes=[39], logprob=0.06900070607662201)]), ChatCompletionTokenLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=1.0, top_logprobs=[TopLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=1.0), TopLogprob(token=' got', bytes=[32, 103, 111, 116], logprob=0.0)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.7065914869308472, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.7065914869308472), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=0.17981767654418945)]), ChatCompletionTokenLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.6476695537567139, top_logprobs=[TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=0.6476695537567139), TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=0.2964670658111572)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=0.21642236411571503, top_logprobs=[TopLogprob(token='!\n\n', bytes=[33, 10, 10], logprob=0.21642236411571503), TopLogprob(token='!', bytes=[33], logprob=0.19332969188690186)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=0.8657481074333191, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=0.8657481074333191), TopLogprob(token=' this', bytes=[32, 116, 104, 105, 115], logprob=0.04993809014558792)]), ChatCompletionTokenLogprob(token='202', bytes=[50, 48, 50], logprob=0.9565137028694153, top_logprobs=[TopLogprob(token='202', bytes=[50, 48, 50], logprob=0.9565137028694153), TopLogprob(token='201', bytes=[50, 48, 49], logprob=0.04348630830645561)]), ChatCompletionTokenLogprob(token='0', bytes=[48], logprob=1.0, top_logprobs=[TopLogprob(token='0', bytes=[48], logprob=1.0), TopLogprob(token='2', bytes=[50], logprob=0.0)])], refusal=None), message=ChatCompletionMessage(content="The Los Angeles Dodgers did not win the World Series in 2020. The Los Angeles Dodgers actually won the World Series in 2020, but I'll explain that later.\n\nHowever, if you're referring to a non-US-based team winning the World Series, then it was the **Los Angeles Dodgers** who won the championship!\n\nThe Dodgers defeated the **San Diego Padres**, and more specifically, they beat the San Diego Padres' counterpart, **the Arizona Diamondbacks**, but not in a game. The winner of the 2020 world series was actually the **Los Angeles Dodgers**.\n\nHowever, I will explain that it's the Dodgers who won the championship in 2020", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1731392381, model='_', object='chat.completion', service_tier=None, system_fingerprint='_', usage=CompletionUsage(completion_tokens=137, prompt_tokens=21, total_tokens=158, completion_tokens_details=None, prompt_tokens_details=None))

OpenAI response:

ChatCompletion(id='chatcmpl-ASeUOBEsi8vV3d8WeJgMo9g5iM69o', choices=[Choice(finish_reason='stop', index=0, logprobs=ChoiceLogprobs(content=[ChatCompletionTokenLogprob(token='The', bytes=[84, 104, 101], logprob=-0.00012356207, top_logprobs=[TopLogprob(token='The', bytes=[84, 104, 101], logprob=-0.00012356207), TopLogprob(token='In', bytes=[73, 110], logprob=-9.000124)]), ChatCompletionTokenLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=-2.188868e-05, top_logprobs=[TopLogprob(token=' Los', bytes=[32, 76, 111, 115], logprob=-2.188868e-05), TopLogprob(token=' ', bytes=[32], logprob=-10.750022)]), ChatCompletionTokenLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=0.0, top_logprobs=[TopLogprob(token=' Angeles', bytes=[32, 65, 110, 103, 101, 108, 101, 115], logprob=0.0), TopLogprob(token=' Angels', bytes=[32, 65, 110, 103, 101, 108, 115], logprob=-25.25)]), ChatCompletionTokenLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.0, top_logprobs=[TopLogprob(token=' Dodgers', bytes=[32, 68, 111, 100, 103, 101, 114, 115], logprob=0.0), TopLogprob(token=' Lakers', bytes=[32, 76, 97, 107, 101, 114, 115], logprob=-18.25)]), ChatCompletionTokenLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.0, top_logprobs=[TopLogprob(token=' won', bytes=[32, 119, 111, 110], logprob=0.0), TopLogprob(token='won', bytes=[119, 111, 110], logprob=-20.375)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.0, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.0), TopLogprob(token=' The', bytes=[32, 84, 104, 101], logprob=-23.625)]), ChatCompletionTokenLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=-0.0046202415, top_logprobs=[TopLogprob(token=' World', bytes=[32, 87, 111, 114, 108, 100], logprob=-0.0046202415), TopLogprob(token=' ', bytes=[32], logprob=-5.37962)]), ChatCompletionTokenLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=0.0, top_logprobs=[TopLogprob(token=' Series', bytes=[32, 83, 101, 114, 105, 101, 115], logprob=0.0), TopLogprob(token='<|end|>', bytes=None, logprob=-23.25)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=0.0, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=0.0), TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=-21.0)]), ChatCompletionTokenLogprob(token=' ', bytes=[32], logprob=0.0, top_logprobs=[TopLogprob(token=' ', bytes=[32], logprob=0.0), TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-20.25)]), ChatCompletionTokenLogprob(token='202', bytes=[50, 48, 50], logprob=0.0, top_logprobs=[TopLogprob(token='202', bytes=[50, 48, 50], logprob=0.0), TopLogprob(token=' ', bytes=[32], logprob=-24.875)]), ChatCompletionTokenLogprob(token='0', bytes=[48], logprob=0.0, top_logprobs=[TopLogprob(token='0', bytes=[48], logprob=0.0), TopLogprob(token='<|end|>', bytes=None, logprob=-23.375)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.078990474, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.078990474), TopLogprob(token=',', bytes=[44], logprob=-2.5789905)]), ChatCompletionTokenLogprob(token=' They', bytes=[32, 84, 104, 101, 121], logprob=-0.02977062, top_logprobs=[TopLogprob(token=' They', bytes=[32, 84, 104, 101, 121], logprob=-0.02977062), TopLogprob(token='<|end|>', bytes=None, logprob=-3.5297706)]), ChatCompletionTokenLogprob(token=' defeated', bytes=[32, 100, 101, 102, 101, 97, 116, 101, 100], logprob=-6.704273e-07, top_logprobs=[TopLogprob(token=' defeated', bytes=[32, 100, 101, 102, 101, 97, 116, 101, 100], logprob=-6.704273e-07), TopLogprob(token=' clin', bytes=[32, 99, 108, 105, 110], logprob=-15.500001)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.0, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=0.0), TopLogprob(token='the', bytes=[116, 104, 101], logprob=-22.4375)]), ChatCompletionTokenLogprob(token=' Tampa', bytes=[32, 84, 97, 109, 112, 97], logprob=0.0, top_logprobs=[TopLogprob(token=' Tampa', bytes=[32, 84, 97, 109, 112, 97], logprob=0.0), TopLogprob(token=' Tamp', bytes=[32, 84, 97, 109, 112], logprob=-20.0)]), ChatCompletionTokenLogprob(token=' Bay', bytes=[32, 66, 97, 121], logprob=0.0, top_logprobs=[TopLogprob(token=' Bay', bytes=[32, 66, 97, 121], logprob=0.0), TopLogprob(token=' Bays', bytes=[32, 66, 97, 121, 115], logprob=-21.75)]), ChatCompletionTokenLogprob(token=' Rays', bytes=[32, 82, 97, 121, 115], logprob=0.0, top_logprobs=[TopLogprob(token=' Rays', bytes=[32, 82, 97, 121, 115], logprob=0.0), TopLogprob(token=' rays', bytes=[32, 114, 97, 121, 115], logprob=-22.6875)]), ChatCompletionTokenLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.030990437, top_logprobs=[TopLogprob(token=' in', bytes=[32, 105, 110], logprob=-0.030990437), TopLogprob(token=' to', bytes=[32, 116, 111], logprob=-4.0309906)]), ChatCompletionTokenLogprob(token=' six', bytes=[32, 115, 105, 120], logprob=-0.00035929924, top_logprobs=[TopLogprob(token=' six', bytes=[32, 115, 105, 120], logprob=-0.00035929924), TopLogprob(token=' a', bytes=[32, 97], logprob=-8.12536)]), ChatCompletionTokenLogprob(token=' games', bytes=[32, 103, 97, 109, 101, 115], logprob=0.0, top_logprobs=[TopLogprob(token=' games', bytes=[32, 103, 97, 109, 101, 115], logprob=0.0), TopLogprob(token=' game', bytes=[32, 103, 97, 109, 101], logprob=-23.1875)]), ChatCompletionTokenLogprob(token=' to', bytes=[32, 116, 111], logprob=-0.009240657, top_logprobs=[TopLogprob(token=' to', bytes=[32, 116, 111], logprob=-0.009240657), TopLogprob(token='.', bytes=[46], logprob=-4.7592406)]), ChatCompletionTokenLogprob(token=' claim', bytes=[32, 99, 108, 97, 105, 109], logprob=-0.4698139, top_logprobs=[TopLogprob(token=' claim', bytes=[32, 99, 108, 97, 105, 109], logprob=-0.4698139), TopLogprob(token=' capture', bytes=[32, 99, 97, 112, 116, 117, 114, 101], logprob=-1.469814)]), ChatCompletionTokenLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.1269281, top_logprobs=[TopLogprob(token=' the', bytes=[32, 116, 104, 101], logprob=-0.1269281), TopLogprob(token=' their', bytes=[32, 116, 104, 101, 105, 114], logprob=-2.126928)]), ChatCompletionTokenLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=-0.04300729, top_logprobs=[TopLogprob(token=' championship', bytes=[32, 99, 104, 97, 109, 112, 105, 111, 110, 115, 104, 105, 112], logprob=-0.04300729), TopLogprob(token=' title', bytes=[32, 116, 105, 116, 108, 101], logprob=-3.1680074)]), ChatCompletionTokenLogprob(token='.', bytes=[46], logprob=-0.0011939177, top_logprobs=[TopLogprob(token='.', bytes=[46], logprob=-0.0011939177), TopLogprob(token=',', bytes=[44], logprob=-6.751194)])], refusal=None), message=ChatCompletionMessage(content='The Los Angeles Dodgers won the World Series in 2020. They defeated the Tampa Bay Rays in six games to claim the championship.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1731392188, model='gpt-4o-2024-08-06', object='chat.completion', service_tier=None, system_fingerprint='fp_159d8341cc', usage=CompletionUsage(completion_tokens=27, prompt_tokens=17, total_tokens=44, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))

nguyenhoangthuan99 commented 1 week ago

The root cause is llamacpp only return confident score for each token, not the log probs. In this pr #289, I've add log before return the result and also add double epsilon value to prevent -inf logprobs

gabrielle-ong commented 5 days ago

Hi @nguyenhoangthuan99, I think there might still be some mathematical issue - it is now returning 2.22 or -36.04 for all logprobs (as in your screenshot, same on my local for different models)

In particular, Im testing this headline "Tennis Champion plays the drums in novel way" - I expecting close to 50/50 between Sports and Arts OpenAI Cortex v1.0.38-13.11.24

nguyenhoangthuan99 commented 20 hours ago

With cortex.llamacpp, we are using the probs output from llamacpp api

const auto* cur_p = common_sampler_get_candidates(slot.smpl);
result.tok = id;

for (size_t i = 0; i < (size_t)slot.sparams.n_probs; ++i) {
        result.probs.push_back({
            cur_p->data[i].id,
            i >= cur_p->size ? 0.0f : cur_p->data[i].p,
        });
      }

So the probs is calculated directly by llamacpp and we just get the result and get log of it.

We can also adjust sampling params to make log probs look better, for example I add the top_p and temperature to chat completion params:

params = {
        "messages": messages,
        "model": model,  # Use the parameter 'model' instead of the global 'MODEL'
        "max_tokens": max_tokens,
        "temperature": temperature,
        "stop": stop,
        "seed": seed,
        "logprobs": logprobs,
        "top_logprobs": top_logprobs,
        "top_p":1,
        "temperature":1
    }

If I run this the log probs look like this

It won't return 0 and 100% all the times

janhq / cortex.llamacpp

feat: [support log prob like OpenAI API] #262

Output:

Code: