`v1/completions` does not include `data: ` prefix when `stream:true`

[josephrocca]

I've just swapped over the endpoints in my code, and the parsing logic broke for streaming responses due to the lack of data: before the JSON object. Is this intended behavior, for some reason?

The only OpenAI v1/completion endpoint that I've tested is text-davinci-003, and it returns a stream where each JSON object is prefixed with data:.

[peakji]

Basaran's response format is identical to OpenAI's, which includes the data: prefix for JSON objects:

# curl "http://127.0.0.1:8888/v1/completions?stream=true&prompt=hi"   
data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"roji","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" and","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" hypoc","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"ot","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"yl","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"\n\n","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"H","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"ydro","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"cot","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":"yl","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" is","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" a","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" genus","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" of","index":0,"logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-dead34f1d82f3e6f72dc469c","object":"text_completion","created":1681644219,"model":"bigscience/bloomz-560m","choices":[{"text":" genera","index":0,"logprobs":null,"finish_reason":"length"}]}

data: [DONE]

[peakji]

Are you using EventSource in browsers? I remember the EventSource API automatically strips the data: prefix as it is part of the event stream specification.

[josephrocca]

@peakji No but I think I might have worked out what's going on: I am using stream:true in the POST request rather than as a URL parameter. Does Basaran support this? I think it might be just returning a normal non-streaming JSON response.

Edit: Yep pretty sure this is what's happening.

[peakji]

Basaran supports passing options in request body, just remember to add the Content-Type: application/json header:

curl -X POST -H "Content-Type:application/json" -d '{"stream":true}' "http://127.0.0.1:8888/v1/completions"

[josephrocca]

Ah you're right that it was the lack of content-type header - might be worth adding an error if the body contains data, but there's no content-type header (unless that is for some reason valid in some cases).