search

explainers-by-googlers / prompt-api

A proposal for a web API for prompting browser-provided language models
Creative Commons Attribution 4.0 International
266 stars 20 forks source link

[bug] Stream outputs the last item 3 times #13

Closed xenova closed 4 months ago

xenova commented 4 months ago

As stated in the title, when streaming, the last item is output 3 times, almost certainly because the tokens generated are special tokens which are stripped away. It would be great if there was a way to set an option like HF transformers skip_special_tokens: false where we can see these tokens being generated. Alternatively, no chunks should be generated for these special tokens.

Example code:

const session = await ai.createTextSession();
const stream = session.promptStreaming('Tell me a joke');
for await (const chunk of stream) {
  console.log(chunk);
}

Examples:

image

image

domenic commented 4 months ago

I think you are reporting an issue against Chromium, and not the explainer. The proper place to do so is https://crbug.com. https://github.com/explainers-by-googlers/prompt-api/blob/main/chrome-implementation-differences.md may also be of interest.

hemanth commented 4 months ago

@domenic what would be your suggestion on markdown stream parser being baked into the stream?