search

hyperonym / basaran

Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
MIT License
1.29k stars 80 forks source link

feat(server): add support for chat completion #101

Closed peakji closed 1 year ago

peakji commented 1 year ago

Checklist

Compatibility

Parameter Basaran OpenAI Default Value Maximum Value
model ○ ● - -
messages ● ● [] CHAT_MAX_PROMPT
min_tokens ● ○ 0 CHAT_MAX_TOKENS
max_tokens ● ● 512 CHAT_MAX_TOKENS
temperature ● ● 1.0 -
top_p ● ● 1.0 -
n ● ● 1 CHAT_MAX_N
stream ● ● false -
stop ○ ● - -
presence_penalty ○ ● - -
frequency_penalty ○ ● - -
logit_bias ○ ● - -
user ○ ● - -

Examples

Chat completion (n=1, stream=false)

{
    "id": "chatcmpl-6z5sqEUkSdUyWqsNFRyyJ1s7kCzpM",
    "object": "chat.completion",
    "created": 1680018644,
    "model": "gpt-3.5-turbo-0301",
    "usage": {
        "prompt_tokens": 12,
        "completion_tokens": 14,
        "total_tokens": 26
    },
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "I was created by a team of developers and engineers at OpenAI."
            },
            "finish_reason": "stop",
            "index": 0
        }
    ]
}

Chat completion (n=2, stream=false)

{
    "id": "chatcmpl-6z5sOxJ318FNEWVkDsRbsZPS7wexL",
    "object": "chat.completion",
    "created": 1680018616,
    "model": "gpt-3.5-turbo-0301",
    "usage": {
        "prompt_tokens": 12,
        "completion_tokens": 34,
        "total_tokens": 46
    },
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "I was created by a team of developers at OpenAI."
            },
            "finish_reason": "stop",
            "index": 0
        },
        {
            "message": {
                "role": "assistant",
                "content": "I was created by a team of developers at OpenAI, using advanced artificial intelligence and natural language processing technologies."
            },
            "finish_reason": "stop",
            "index": 1
        }
    ]
}

Chat completion (n=1, stream=true)

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"I"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" was"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" created"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" by"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" a"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" team"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" of"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" developers"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" at"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" Open"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"AI"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"."},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5sk5NJqhsESrfo25sQzjMfCbcZ0","object":"chat.completion.chunk","created":1680018638,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]

Chat completion (n=2, stream=true)

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"I"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"I"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" was"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" was"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" created"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" created"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" by"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" by"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" a"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" Open"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"AI"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" team"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"."},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" of"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" programmers"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" and"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" developers"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" at"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":" Open"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"AI"},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"."},"index":1,"finish_reason":null}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: {"id":"chatcmpl-6z5rT81CjeF0YZH9BCf5fXscq2iqA","object":"chat.completion.chunk","created":1680018559,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":1,"finish_reason":"stop"}]}

data: [DONE]

Text completion (n=1, stream=false)

{
    "id": "cmpl-6z60oeEN6Wf4O3Ad0OvAMtA92ELQd",
    "object": "text_completion",
    "created": 1680019138,
    "model": "text-davinci-003",
    "choices": [
        {
            "text": "\n\nThis is indeed a test",
            "index": 0,
            "logprobs": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 5,
        "completion_tokens": 7,
        "total_tokens": 12
    }
}
codecov-commenter commented 1 year ago

Codecov Report

Patch and project coverage have no change.

Comparison is base (d219972) 94.23% compared to head (72a1eaa) 94.23%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #101 +/- ## ======================================= Coverage 94.23% 94.23% ======================================= Files 7 7 Lines 312 312 ======================================= Hits 294 294 Misses 18 18 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperonym). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=hyperonym)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.