on_chat_start function called if background task takes too long

[frankwanicka]

While processing on_message, I am doing some background processing. If it takes too long (~30 seconds), the function decorated by on_chat_start gets called again and so the app basically resets. Is there a timeout configured somewhere in the chainlit code or further down in asyncio or something that would cause that? I have dug around in the source code a bit and haven't found the smoking gun yet.

[willydouhard]

Hello, does that long processing need to happen for every user? Otherwise you can run that at the beginning of your python file and use the result in on_chat_start. You could also cache the result so that it would work nicely with the file watcher while developing.

[frankwanicka]

The long processing is as the result of a prompt by an individual user. It is dynamic, so it can't be run in on_chat_start.

[willydouhard]

Okay, I will investigate to check if this is a timeout issue

[kaonick]

I also encountered the same problem. I need to be able to set the timeout mechanism. When encountering a long-term execution situation, need to extend the timeout time

[willydouhard]

Can you show me what the long running function looks like? Is it synchronous or asynchronous?

[frankwanicka]

It's a lot of code, that I can't share directly, but I'll describe it. I started with the openai-functions example from the chainlit cookbook repo. Instead of calling the synchronous get_current_weather function, I am calling an asynchronous function that can do various things depending on the current state. The two places where it currently times out is a call to the OpenAI API using the gpt-4 model with a large prompt and a large response. The other is is a path that makes a dozen or more calls sequentially to the ElevenLabs API.

The common denominator is the amount of time it takes. Everything works fine if I reduce the prompt and response size in the call to OpenAI and limit it to only a few calls to ElevenLabs so it completes in under 15 seconds.

[willydouhard]

I was trying to reproduce with a minimal code:

import chainlit as cl

@cl.on_chat_start
async def start():
    print("START")
    await cl.sleep(40)
    print("DONE")
    await cl.Message(content="Ok").send()

This works on my machine.

Even the synchronous blocking version works:

import chainlit as cl
import time

@cl.on_chat_start
async def start():
    print("START")
    time.sleep(40)
    print("DONE")
    await cl.Message(content="Ok").send()

Could try running those snippets and see what happens?

[frankwanicka]

Right after my previous response, I tried to reproduce what I am seeing by converting the get_current_weather function to async and having it sleep, but that does not cause it. I'll see if I can create a simple way to reproduce it.

[frankwanicka]

So, this reproduces the issue for me consistently. Sorry it's so long. I am pressed for time today. It should start by prompting "What would you like to create today?" Answer "A podcast script about the Beatles". For me, after it processes for a bit, it re-prompts "What would you like to create today?" since on_chat_start gets called again.

import openai
import ast
import os
import chainlit as cl

from langchain import PromptTemplate
from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain

openai.api_key = os.environ.get("OPENAI_API_KEY")

MAX_ITER = 5

async def generate_podcast_script():
    llm = ChatOpenAI(model_name="gpt-4-0613", temperature=0.7)
    participants = """
    John is the host and has a neutral stance
    Paul is a guest and has a positive stance
    George is a guest and has a negative stance
    Ringo is a guest and has a neutral stance
    """

    outline = """
I. Introduction
A. Sir Paul McCartney's use of artificial intelligence to create a final Beatles song

II. Background on the song
    A. 1978 Lennon composition called Now And Then
    B. Considered as a possible reunion song in 1995
    C. Demo received by Sir Paul from Yoko Ono

III. Previous attempts to record the song
    A. Free As A Bird and Real Love completed and released in 1995 and 96
    B. Session for Now And Then quickly abandoned

IV. Technical issues and speculation
    A. Issues with original recording, including persistent "buzz"
    B. Speculation on stolen recording released in 2009

V. Sir Paul's desire to finish the song
    A. Repeatedly expressed desire to finish the song
    B. Collaboration with Jeff Lynne mentioned

VI. Technology enabling completion of the song
    A. Peter Jackson's Get Back documentary and AI separation of voices
    B. Sir Paul's ability to "duet" with Lennon on recent tour
    C. Creation of new surround sound mixes of Revolver album

VII. Concerns and excitement about AI
    A. Sir Paul's concern about AI being used to create fake recordings
    B. Acknowledgment of the future and uncertainty surrounding AI

VIII. Conclusion
    A. Mention of Sir Paul's new book and photography exhibition at the National Portrait Gallery
"""

    scriptTemplate = f"""You are an assistant that helps write a script for a podcast based on an outline.
    Answer the prompt based on the outline and article below.

    Participants:
    {{participants}}

    Outline:
    {{outline}}

    Prompt: {{query}}

    Answer: """

    query = f"From the outline write a long, detailed script for a podcast. The podcast will have participants in roles with perspectives. Each line in the script needs to start with only the first name of who is talking."

    scriptChain = LLMChain(llm=llm, prompt=PromptTemplate.from_template(scriptTemplate))
    scriptResult = scriptChain(
        inputs={
            "query": query,
            "participants": participants,
            "outline": outline,
        },
        return_only_outputs=True,
    )

    return scriptResult["text"]

functions = [
    {
        "name": "generate_podcast_script",
        "description": "Generate a podcast script",
        "parameters": {
            "type": "object",
            "properties": {
                "subject": {
                    "type": "string",
                    "description": "The subject matter for the script",
                },
            },
            "required": ["subject"],
        },
    }
]

@cl.on_chat_start
async def start_chat():
    print("starting chat")
    cl.user_session.set(
        "message_history",
        [{"role": "system", "content": "You are a helpful assistant."}],
    )

    await cl.Message(
        author="Assistant",
        content="What would you like to create today?",
    ).send()

@cl.on_message
async def run_conversation(user_message: str):
    message_history = cl.user_session.get("message_history")
    message_history.append({"role": "user", "content": user_message})

    cur_iter = 0

    while cur_iter < MAX_ITER:
        response = await openai.ChatCompletion.acreate(
            model="gpt-3.5-turbo-0613",
            messages=message_history,
            functions=functions,
            function_call="auto",
        )

        message = response["choices"][0]["message"]
        await cl.Message(author=message["role"], content=message["content"]).send()
        message_history.append(message)

        if not message.get("function_call"):
            break

        function_name = message["function_call"]["name"]
        # arguments = ast.literal_eval(message["function_call"]["arguments"])

        await cl.Message(
            author=function_name,
            content=str(message["function_call"]),
            language="json",
            indent=1,
        ).send()

        function_response = await generate_podcast_script()

        message_history.append(
            {
                "role": "function",
                "name": function_name,
                "content": function_response,
            }
        )

        await cl.Message(
            author=function_name,
            content=str(function_response),
            language="json",
            indent=1,
        ).send()

        cur_iter += 1

[frankwanicka]

I'm sure I can come up with one that doesn't require an OpenAI API key later today if needed. Note that the API does fully complete. It's not failing and throwing an error and causing a reset.

[willydouhard]

Was able to reproduce, the issue is when you call your LLMChain. Just replace the invocation by:

   scriptResult = await scriptChain.acall(
        inputs={
            "query": query,
            "participants": participants,
            "outline": outline,
        },
        return_only_outputs=True,
    )

You should never call a long running agent synchronously. If it supports async (which is the case here) call the async implementation and await it. If it doesn't you can use make_async.

Works like a charm now!

Btw you have good music taste

[willydouhard]

You can further improve it by passing a callback handler and enable streaming!

async def generate_podcast_script():
    llm = ChatOpenAI(model_name="gpt-4-0613", streaming=True, temperature=0.7)
    participants = """
    John is the host and has a neutral stance
    Paul is a guest and has a positive stance
    George is a guest and has a negative stance
    Ringo is a guest and has a neutral stance
    """

    outline = """
I. Introduction
A. Sir Paul McCartney's use of artificial intelligence to create a final Beatles song

II. Background on the song
    A. 1978 Lennon composition called Now And Then
    B. Considered as a possible reunion song in 1995
    C. Demo received by Sir Paul from Yoko Ono

III. Previous attempts to record the song
    A. Free As A Bird and Real Love completed and released in 1995 and 96
    B. Session for Now And Then quickly abandoned

IV. Technical issues and speculation
    A. Issues with original recording, including persistent "buzz"
    B. Speculation on stolen recording released in 2009

V. Sir Paul's desire to finish the song
    A. Repeatedly expressed desire to finish the song
    B. Collaboration with Jeff Lynne mentioned

VI. Technology enabling completion of the song
    A. Peter Jackson's Get Back documentary and AI separation of voices
    B. Sir Paul's ability to "duet" with Lennon on recent tour
    C. Creation of new surround sound mixes of Revolver album

VII. Concerns and excitement about AI
    A. Sir Paul's concern about AI being used to create fake recordings
    B. Acknowledgment of the future and uncertainty surrounding AI

VIII. Conclusion
    A. Mention of Sir Paul's new book and photography exhibition at the National Portrait Gallery
"""

    scriptTemplate = f"""You are an assistant that helps write a script for a podcast based on an outline.
    Answer the prompt based on the outline and article below.

    Participants:
    {{participants}}

    Outline:
    {{outline}}

    Prompt: {{query}}

    Answer: """

    query = f"From the outline write a long, detailed script for a podcast. The podcast will have participants in roles with perspectives. Each line in the script needs to start with only the first name of who is talking."

    scriptChain = LLMChain(llm=llm, prompt=PromptTemplate.from_template(scriptTemplate))
    scriptResult = await scriptChain.acall(
        inputs={
            "query": query,
            "participants": participants,
            "outline": outline,
        },
        return_only_outputs=True,
        callbacks=[cl.AsyncLangchainCallbackHandler()],
    )

    return scriptResult["text"]

You will be able to see what happens in real time before the final answer is returned

[willydouhard]

Also the while loop if probably too much here, you can get rid of it imo

[frankwanicka]

Got it. Thanks very much. I was hoping it was a simple mistake on my part.

[Keramatfar]

print("START")
    time.sleep(40)
    print("DONE")
    await cl.Message(content="Ok").send()

with time.sleep(50) I always does not receive the responce.

[Keramatfar]

https://github.com/Chainlit/chainlit/issues/803