OpenAI Assistants V2 Stream Error After Function Call

[jackitaliano]

Scenario

• deep-chat-dev v9.0.179
• directConnection to OpenAI Assistants V2
• streaming enabled

When the assistant first provides text, then makes an async function call, then provides text again (within the same run), I'm getting an error:

Error: No valid stream events were sent.
Make sure the events are using {text: string} or {html: string} format.
You can also augment them using the responseInterceptor property: https://deepchat.dev/docs/interceptors#responseInterceptor
    at Re.finaliseStreamedMessage (deepChat.js:3291:36)
    at Re.newMessage (deepChat.js:3306:14)
    at L._callee94$ (deepChat.js:9066:151)
    at tryCatch (deepChat.js:18:1062)
    at Generator.<anonymous> (deepChat.js:18:3008)
    at Generator.next (deepChat.js:18:1699)
    at asyncGeneratorStep$1 (deepChat.js:19:103)
    at _next (deepChat.js:20:194)
    at deepChat.js:20:364
    at new Promise (<anonymous>)

Recreate

• Ask the assistant to perform a task that is clearly defined by a provided function, but to explain what it's going to do prior to calling the function. Then to call the function without confirming.
• Must be an async function that does not immediately return.

Thoughts

I've tried debugging as much as I can, and as far as I can tell I've not done anything in the response interceptor to cause this. Our responseInterceptor is called, but does not change the object at all in this scenario.

I've also stepped through deep-chat execution, and it seems this._elements is undefined after an async tool call in finaliseStreamedMessage. Kind of curious because stepping through shows it as undefined, logging this has a defined _elements field, and logging this._elements is undefined. Thinking this has something to do with why it only occurs on async calls that don't return immediately, which would affect the results based on timing?

[OvidijusParsiunas]

Hey @jackitaliano.

Thankyou for letting me know about the issue. I apologise but I am unable to fully understand on how to re-produce the error. I have attempted to use the default Assistant Functions example and it appeared to work fine.

Could you let me know what code to use and what exactly to prompt so I can reproduce it on my computer.

Thankyou very much!!

[jackitaliano]

Define assistant function as:

{
  "name": "text_to_image",
  "description": "Generate an image from a text description.",
  "parameters": {
    "type": "object",
    "properties": {
        "prompt": {
          "type": "string",
          "description": "The description of the image to generate."
        }
    },
    "required": []
  }
}

Define function as:

async function genImage() {
  await new Promise((resolve) => setTimeout(resolve, 500));
  return "The image has been inserted into the chat."
}

Send message to assistant: "Generate an image of a military aircraft. First tell me the exact prompt, then generate the image. Then tell me about the image."

It seems to cause this error with any async function, so you could also do the example ones like "get_weather", just with a timeout. This is just the parts of one I had already.

Edit: In the Assistant Functions examples, that might be:

// using JavaScript for a simplified example

chatElementRef.directConnection = {
  openAI: {
    assistant: {
      assistant_id: 'placeholder-id',
      function_handler: (functionsDetails) => {
        return functionsDetails.map(async (functionDetails) => this.getCurrentWeather(functionDetails.arguments));
      },
    },
    key: 'placeholder-key',
  },
};

async function getCurrentWeather(location) {
  await new Promise((resolve) => setTimeout(resolve, 500));
  location = location.toLowerCase();
  if (location.includes('tokyo')) {
    return 'Good';
  } else if (location.includes('san francisco')) {
    return 'Mild';
  } else {
    return 'Very Hot';
  }
}

[OvidijusParsiunas]

Hi @jackitaliano.

Thankyou for providing the info to reproduce the error. I am unfortunately going on a week long trip with limited access to the internet, hence I will unfortunately not be able to take care of the problem until I get back. My apologies. I will update this thread as soon as I am able to get progress on this.

[jackitaliano]

Hi @jackitaliano.

Thankyou for providing the info to reproduce the error. I am unfortunately going on a week long trip with limited access to the internet, hence I will unfortunately not be able to take care of the problem until I get back. My apologies. I will update this thread as soon as I am able to get progress on this.

No worries, have a good trip!

[OvidijusParsiunas]

Wanted to give you an update! I was able to fix the error for the default example, however I am still getting an error for genImage. I will continue to investigate this tomorrow.

[OvidijusParsiunas]

The fix has now been added to deep-chat-dev and deep-chat-react-dev package versions 9.0.183. Let me know if this works for you. Thanks!

[OvidijusParsiunas]

Hi, the fixes mentioned in this issue is now available in Deep Chat version 2.0.0.