feat: [support stream_option for OpenAI API compatible]

janhq / cortex.llamacpp

cortex.llamacpp is a high-efficiency C++ inference engine for edge computing. It is a dynamic library that can be loaded by any server at runtime.

GNU Affero General Public License v3.0

22 stars 3 forks source link

Closed nguyenhoangthuan99 closed 1 week ago

nguyenhoangthuan99 commented 3 weeks ago

Problem

field: stream_options
object or null
Optional Defaults to null. Options for streaming response. Only set this when you set stream: true. {"include_usage": boolean } Optional If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array. All other chunks will also include a usage field, but with a null value.

gabrielle-ong commented 1 week ago

✅ QA: on cortex.llama-cpp v0.1.37-01.11.24 API request: v1/chat/completions

"stream_options": {
        "include_usage": false | true
},