Lots of HTML output

[motaatmo]

The bug When calling guidance using an openai model (GPT4o-mini; I didn't try other models yet), lots of output are created, that clutter my own output. It looks HTMLish:

("<||_html:<div style='display: flex; border-bottom: 1px solid rgba(127, 127, "
 "127, 0.2);  justify-content: center; align-items: center;'><div style='flex: "
 "0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; "
 'padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: '
 "pre-wrap; margin-bottom: 0px;'>_||>")
("<||_html:<div style='display: flex; border-bottom: 1px solid rgba(127, 127, "
 "127, 0.2);  justify-content: center; align-items: center;'><div style='flex: "
 "0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; "
 'padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: '
 "pre-wrap; margin-bottom: 0px;'>_||><||_#NODISP_||><|im_start|>system\n"

To Reproduce

import guidance

model = guidance.models.OpenAI(
    "gpt-4o-mini-2024-07-18",
    api_key=(
        "XXXXXXXXXXXXXXXXXXXX"
    ))

with guidance.user():
    model += "Das ist ein test"

System info (please complete the following information):

• OS: Debian
• Guidance Version: 0.1.16

[hudson-ai]

Hey @motaatmo thanks for the issue. What context are you running this code in? Jupyter? ipython at the shell? The standard python repl?

[motaatmo]

I called the script using standard python (non-interactive, from the command line)

Hudson Cooper @.***> schrieb am Mo., 16. Sept. 2024, 21:18:

Hey @motaatmo https://github.com/motaatmo thanks for the issue. What context are you running this code in? Jupyter? ipython at the shell? The standard python repl?

— Reply to this email directly, view it on GitHub https://github.com/guidance-ai/guidance/issues/1024#issuecomment-2353725388, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACO5DS2TBOIAMDQOQ722QYDZW4VJNAVCNFSM6AAAAABOHSFLNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJTG4ZDKMZYHA . You are receiving this because you were mentioned.Message ID: @.***>

[hudson-ai]

Got it. By default, models are quite verbose (and the experience is optimized for Jupyter notebooks -- the command line needs some love...).

For now, you should be able instantiate the model with echo=False to suppress this verbose output:

model = guidance.models.OpenAI(
    "gpt-4o-mini-2024-07-18",
    api_key=(
        "XXXXXXXXXXXXXXXXXXXX"
    ),
    echo=False,
)

[motaatmo]

Works perfectly, thank you!

On Mon, Sep 16, 2024 at 9:54 PM Hudson Cooper @.***> wrote:

Got it. By default, models are quite verbose (and the experience is optimized for Jupyter notebooks -- the command line needs some love...).

For now, you should be able instantiate the model with echo=False to suppress this verbose output:

model = guidance.models.OpenAI( "gpt-4o-mini-2024-07-18", api_key=( "XXXXXXXXXXXXXXXXXXXX" ), echo=False, )

— Reply to this email directly, view it on GitHub https://github.com/guidance-ai/guidance/issues/1024#issuecomment-2353791990, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACO5DS54RA3BTHIWONJNENLZW4ZO7AVCNFSM6AAAAABOHSFLNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJTG44TCOJZGA . You are receiving this because you were mentioned.Message ID: @.***>

[ramchennuru]

My code: from guidance import models, gen, user, assistant import guidance

@guidance **def response(lm, prompt): with user(): lm += prompt

with assistant():
    lm += gen("""\
    ```json
    {{
        "name": "{{name}}",
        "age": "{{age}}",
        "occupation": "{{occupation}}"
    }}
    ```""")**

return lm

def main(): prompt = f""" Give any person name, age, and occupation, example: {{ "name":"John Doe", "age": 30, "occupation": "Software Engineer" }} """

**llm = models.AzureOpenAI(
    model='gpt-4',
    azure_deployment="gpt-4",
    azure_endpoint="",
    api_key=,,
    echo=False,
)**

**lm = llm + response(prompt)

print(lm)**

Check if this script is being run as the main program

if name == "main": main()

Output: **<|im_start|>user

    Give any person name, age, and occupation,
    example:
    {
        "name":"John Doe",
        "age": 30,
        "occupation": "Software Engineer"
    }
<|im_end|>

<|im_start|>assistant { "name":"Emma Smith", "age": 25, "occupation": "Graphic Designer" }<|im_end|>**

@hudson-ai ,Iam still getting the HTML code in output even I add echo=False also.

[hudson-ai]

@ramchennuru sorry that you're getting unexpected output! I'm not currently able to reproduce on my end. I see the output of your print statement (slightly different response, but basically the same):

<|im_start|>user

    Give any person name, age, and occupation,
    example:
    {
        "name":"John Doe",
        "age": 30,
        "occupation": "Software Engineer"
    }
    <|im_end|>
<|im_start|>assistant
{
    "name":"Jane Smith",
    "age": 45,
    "occupation": "Graphic Designer"
}<|im_end|>

Can you paste a snippet here of the HTML you are seeing? It might help dig into the cause.

Also, are you able to test your code using a different model (e.g. just straight OpenAI or a local model)? Doing so might help determine whether the problem is with the AzureOpenAI class itself or if something more is going on. Thanks :)

[rnybergh]

I can confirm the same <|im_start|> style output when using the OpenAI models directly (both gpt-4o-mini and gpt-4o)

[hudson-ai]

Hey @rnybergh, thanks for chiming in!

The original issue concerned HTML that was being printed as a side effect of model generation, not HTML content in the model's output itself (what you see when you do str(lm), print(lm), or access a named output like lm[key]).

The <|im_start|>, etc. "tags" that you are seeing aren't actually HTML -- they are just stringified representations of the "role" (system, user, or assistant) that a given block of text falls into. If you were running a local chat model, this is actually the form that a list of "message" objects has to be converted to in order for the underlying completion model to work. As such, these are considered "expected output" rather than evidence of a bug.

@ramchennuru can you confirm whether the role tags are what you were referring to when you said you were getting HTML in your output?

If either of you have expectations about what model output should look like that disagree with what you are seeing here, I would recommend opening a separate "Feature Request" issue that includes some examples of what you'd like to see instead. Thank you!