Highlight logprob with ChatInterface

[ahuang11]

Based on https://cookbook.openai.com/examples/using_logprobs

Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is log(p), where p = probability of a token occurring at a specific position based on the previous tokens in the context. Some key points about logprobs

Now, with logprobs enabled, we can see exactly how confident the model is in its predictions, which is crucial for creating an accurate and trustworthy classifier.

Lighter colors indicate lower probability, darker colors / black indicate higher probability

import panel as pn
import numpy as np
from math import exp
from openai import AsyncOpenAI

pn.extension()

COLORS = [
    '#94c4df',
    '#1764ab',
    '#084a92',
    '#08306b',
    '#000000'
]

def highlight(text, log_prob):
    linear_prob = np.round(exp(log_prob) * 100, 2)
    if linear_prob >= 0 and linear_prob < 20:
        color_idx = 0
    elif linear_prob >= 20 and linear_prob < 40:
        color_idx = 1
    elif linear_prob >= 40 and linear_prob < 60:
        color_idx = 2
    elif linear_prob >= 60 and linear_prob < 80:
        color_idx = 3
    else:
        color_idx = 4

    # Generate HTML output with the chosen color
    if "'" in text:
        html_output = f'<span style="color: {COLORS[color_idx]};">{text}</span>'
    else:
        html_output = f"<span style='color: {COLORS[color_idx]}'>{text}</span>"
    return html_output

async def callback(contents: str, user: str, instance: pn.chat.ChatInterface):
    messages.append({"role": "user", "content": contents})
    response = await aclient.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=messages,
        stream=True,
        logprobs=True,
        temperature=1.9
    )
    message = ""
    async for chunk in response:
        part = chunk.choices[0].delta.content
        token = chunk.choices[0].logprobs
        if part and token:
            log_prob = token.content[0].logprob
            message += highlight(part, log_prob)
            yield message
    messages.append({"role": "user", "content": message})

messages = []
aclient = AsyncOpenAI()
chat_interface = pn.chat.ChatInterface(callback=callback, callback_user="ChatGPT", callback_exception="verbose")
chat_interface.show()

Default temperature

https://github.com/holoviz-topics/panel-chat-examples/assets/15331990/50563b28-fbf3-47e9-a690-cfac290ebde3

Temperature = 1.9

https://github.com/holoviz-topics/panel-chat-examples/assets/15331990/4beeda01-6115-4d6a-b9eb-f728446f595c