Svelte useAssistant stop function does not cancel run

[andrewrisse]

Description

When using the stop function from useAssistant, the stream stops to the client, but the run continues. If you try to start another run, and the first one (the one you thought you cancelled) has not finished, you get the error: "Can't add messages to thread_xyz while a run run_abc is active.", The stop function should also cancel the run: https://platform.openai.com/docs/api-reference/runs/cancelRun

Code example

const {
    status,
    input,
    messages,
    submitMessage,
    stop,
    error
  } = useAssistant({...})

const handleCancel = () => {
 if ($status === 'in_progress') stop();
}

Additional context

The 'error' also has a 'DOMException: BodyStreamBuffer was aborted' error when calling the stop function, but this error is expected and should be ignored.

Side note -If the useAssistant messages included more information about the message (for example, the assistant_id, or createdAt fields) this could also be super helpful.

[jaycoolslm]

Good spot, I believe the issue will be the same for the React lib too (haven't validated).

My initial guess is that there may be something that can be done on the /api endpoint side of things though to handle this when the stream is aborted

ie. you create your own logic which pings a cancel endpoint which on abort

[tomfuller71]

Anyone figured out how to solve this? I’m getting same error with React when trying to submit a new message to a thread after calling stop on active run. In my api route I’m passing along the abortcontroller signal from the request made by useAssistant on the client side in line with the example code. The run.status of cancelling and cancelled are not exposed via the useAssistant hook which just continues to show an ‘awaiting-message’ status after calling stop.

[tomfuller71]

Until a run reach a terminal state 'expired | completed | failed | incomplete | cancelled ' the current thread is locked and no messages can be added to it. Calling stop is not guaranteed to cancel a run - it just aborts the in-flight fetch - so the thread can remain locked until it times out. There is a separate openai cancelRun end point to cancel a run,this also is not guaranteed to cancel ... but so far it has worked for me - but I don't know how to trap if it fails. My hacked together solution was to 1) capture the current run id when creating a new stream, and then have a new DELETE route on my \api\assistant. e.g.

let currentRunId: string | null = null;

export async function POST(req: Request) {
  ...

  return AssistantResponse(
    { threadId, messageId: createdMessage.id },
    async ({ forwardStream }) => {
      // Run the assistant on the thread
      const runStream = openai.beta.threads.runs.stream(threadId, {
        assistant_id:
          assistantId ??
          (() => {
            throw new Error('ASSISTANT_ID is not set');
          })(),
      }).on('event', (params: any) => {
        const { event, data } = params;

        if (event === 'thread.run.created') {
          const { id } = data;
          currentRunId = id;
          console.log('Run created:', id);
        }
      });

      let runResult = await forwardStream(runStream);
    },
  );
}

// Cancel the current run
export async function DELETE(req: Request) {
    const input: { threadId: string | null } = await req.json();
  if (currentRunId && input.threadId) {
    await openai.beta.threads.runs.cancel(input.threadId, currentRunId);
    currentRunId = null;
  }
  return new Response(null, { status: 204 });
}

It feels like there should be a better way to handle cancellations ...