explainers-by-googlers / prompt-api

A proposal for a web API for prompting browser-provided language models
Creative Commons Attribution 4.0 International
267 stars 20 forks source link

Aborting prompt #21

Closed horo-t closed 4 months ago

horo-t commented 4 months ago

prompt() takes some time to execute, and I think there needs to be a way to abort the prompt to free up resources used for it.

I think this can be done by passing an AbortSignal to prompt() and promptStreaming().

dictionary AITextSessionPromptOptions {
  AbortSignal? signal;
};

[Exposed=(Window,Worker)]
interface AITextSession {
  Promise<DOMString> prompt(DOMString input,
                            optional AITextSessionPromptOptions options = {});
  ReadableStream promptStreaming(DOMString input,
                                 optional AITextSessionPromptOptions options = {});
  ...
};
domenic commented 4 months ago

This sounds very reasonable to me.

Note that this shouldn't be strictly necessary, as you can use promptStreaming(), which should immediately return a ReadableStream, and then call stream.cancel().

But AbortSignal integration is idiomatic and can lead to cleaner code, for example when your app already has an AbortSignal it's using for other things. And it's quite convenient for the non-streaming version. So we should add this.

hemanth commented 4 months ago

+1 I came here to post the same.