Streaming is not working with examples.

johnd0e commented 10 months ago

None of following examples produces streaming:

python example_bard.py true
python example_chatgpt1.py true
python example_chatgpt2.py true
python example_claude.py true

P.S. Same with direct curl requests.

Amm1rr commented 10 months ago

It's now fixed and available right now. Just request a pull.

P.S. Claude Stream still needs a little more time to implement.

johnd0e commented 10 months ago

Unfortunately I cannot confirm the fix. Streaming still does not work neither with Bard, nor with Chatgpt

Amm1rr commented 10 months ago

Can you tell me what your output is? Do you pull the latest updates from the repository? Because the released tag is still old.

johnd0e commented 10 months ago

I'm on commit d6243237101e11bcd2517b3744c7dfe2bc07010a, I have pulled it with git, like you said. With endpoints /bard, /claude, /chatgpt, and /claude/v1/chat/completions behavior is the same - I send request, wait (significal delay) and get (pretty big) response in one piece, without streaming.

E.g. my request:

@curl --request POST ^
      --url http://127.0.0.1:8000/chatgpt ^
      --header "accept: application/json" ^
      --header "content-type: application/json" ^
      -T completions.json

    {
      "session_id": "",
      "message": "Sing a song of Fire and Ice",
      "stream": true
    }

The response:

(Verse 1) In a realm of fire and ice, Where dragons soar and direwolves run, Lives entwined in a web of fate, Kings and queens, battles to be won. (Chorus) Oh, a song of fire and ice, Lands of intrigue, honor, and sacrifice, From Winterfell to King's Landing's height, The game of thrones, a dangerous dice. (Verse 2) Winter's breath, a chilling grasp, Across the Wall, the Night King looms, Jon and Daenerys take their stand, As destiny awakens and dooms. (Chorus) Oh, a song of fire and ice, Lands of intrigue, honor, and sacrifice, From Winterfell to King's Landing's height, The game of thrones, a dangerous dice. (Bridge) Schemes and plots, like shadows cast, Lives are shattered, futures collide, Loyalties tested in battles vast, In this realm where hope and darkness ride. (Chorus) Oh, a song of fire and ice, Lands of intrigue, honor, and sacrifice, From Winterfell to King's Landing's height, The game of thrones, a dangerous dice. (Outro) So let the dragons soar up high, And let the wolves roam free and wild, In this world of fire and ice, A symphony of a tale beguiled.

In case of /v1/chat/completions endpoint the things are also incorrect.

```json {"id": "chatcmpl-1692826683.6999755", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ""}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7079394", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "Sure"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7079394", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7239099", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " here"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7354522", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "'s"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.741301", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " a"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7423494", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " short"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7433467", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " verse"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7484002", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " inspired"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.75754", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " by"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.7769613", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " \""}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.857627", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "A"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8586755", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " Song"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8596728", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8606348", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " Ice"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8606348", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.867025", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " Fire"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8680227", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "\":\n\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8680227", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "In"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8690214", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " realms"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8690214", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.8700216", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " fire"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.871021", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.871021", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " ice"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9180522", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " so"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.919057", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " bold"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.920056", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.920056", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "Where"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9210505", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " secrets"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9220507", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " lie"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.923049", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.923049", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " tales"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9240606", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " unfold"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9240606", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9250846", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "Drag"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9260478", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "ons"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9290535", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " soar"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9290535", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9300823", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " wolves"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9310813", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " do"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9310813", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " how"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.932053", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "l"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9330432", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.934059", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "A"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.934059", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " saga"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.935057", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.936078", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " power"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.936078", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.937078", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " a"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9380555", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " f"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.93905", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "ateful"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9400764", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " pro"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9452806", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "wl"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.9796422", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ".\n\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826683.995443", "object": "chat.completion.chunk", "created": 1692826683, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "King"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.0132163", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "dom"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.0333722", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "s"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.0636837", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " clash"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.0897584", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " with"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.1081305", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " honor"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.1091297", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "'s"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.1406307", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " crest"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.1618109", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.1738403", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "While"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.1963382", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " winter"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.2225857", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "'s"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.2416327", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " chill"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.2580607", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " puts"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.2790759", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " realms"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.2941577", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " to"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.326609", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " test"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.3901362", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.3956425", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "L"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.402531", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "ords"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.409465", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.409465", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " ladies"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.4307287", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.4417355", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " a"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.4648762", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " game"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.4839964", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.513791", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " th"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.5294602", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "rones"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.5507648", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.571243", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "In"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.5882065", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "tr"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.6295276", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "igue"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.6354828", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.653588", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " battles"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.667438", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.6852314", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " the"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.7165067", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " rightful"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.7322962", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " owns"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.7490726", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ".\n\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.766971", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "So"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.784101", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " sing"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.8067772", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " this"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.8237715", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " song"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.879157", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.879157", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " fire"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.882177", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.8953595", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " ice"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.9093375", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.9307141", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "A"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826684.960526", "object": "chat.completion.chunk", "created": 1692826684, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " world"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.0198474", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.0208776", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " wonder"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.0208776", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.028266", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " of"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.049727", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " sacrifice"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.0649152", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.0854084", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "Heroes"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.1143394", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " rise"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.1192694", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.1322935", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " and"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.1669304", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " villains"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.1713398", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " scheme"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.1895864", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ",\n"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.2046354", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "In"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.2191033", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " this"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.2362113", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " epic"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.254476", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " tale"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.3090909", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ","}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.312469", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " a"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.316191", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " realm"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.3364224", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "'s"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.3383985", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " grand"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.3527844", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": " dream"}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.3679576", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": "."}, "index": 0, "finish_reason": null}]} {"id": "chatcmpl-1692826685.3896654", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ""}, "index": 0, "finish_reason": "stop"}]} {"id": "chatcmpl-1692826685.3896654", "object": "chat.completion.chunk", "created": 1692826685, "model": "text-davinci-002-render-sha", "choices": [{"delta": {"role": "assistant", "content": ""}, "index": 0, "finish_reason": "stop"}]} ```

Here response seems correct, but again, I get all these messages at once, after significant time.

'Streaming' means that I receive response(s) chunk by chunk, immediately when they ready.

Amm1rr commented 10 months ago

Interesting, Are you trying streaming with examples?

python examples_chatgpt1.py True

johnd0e commented 10 months ago

Of course I did, with same effect.

Amm1rr commented 10 months ago

Actually I don't have access to my system for a couple days, but I think we should start debugging this by looking at the Python version and environment.

What Python version are you using?

johnd0e commented 10 months ago

Python 3.10.11

Amm1rr commented 10 months ago

Can you please share a screenshot of the output results in the API Docs:

http://127.0.0.1:8000/docs

Try with false and true for streaming for ChatGPT or Bard.

johnd0e commented 10 months ago

I have tried both chatgpt and bard, and here are my observations.

with "stream": true both outputs plain text, with new lines
with "stream": false:
- chatgpt outputs quoted text, with quotes displayed as \" and newlines as \n
- bard additionally wraps output in json: {"choices":[{"message":{"content":["...

Thus:

"stream": false seems not useful at all (why may we need such quoted text?)
"stream": true output is good, but still - it appears all in one time. Not streamed.

Amm1rr commented 10 months ago

Thank you John There was a silly mistake in the main file that used "return" instead of "yield".

I can't test the result right now, I just made changes on the mobile phone editor and pushed a commit, but there should be no problem.

Just submitting a pull request for this fix; For Bard and ChatGPT, I'm not sure if Claude is working properly with this fix.

johnd0e commented 10 months ago

There was a silly mistake in the main file that used "return" instead of "yield".

At the moment there are many such places in code, preventing server from launching. And it is not enough to replace there all occurrences of "return" with "yield", because it lead to runtime error during request:

←[31mERROR←[0m:    Exception in ASGI application
Traceback (most recent call last):
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\encoders.py", line 230, in jsonable_encoder
    data = dict(obj)
TypeError: 'async_generator' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\encoders.py", line 235, in jsonable_encoder
    data = vars(obj)
TypeError: vars() argument must have __dict__ attribute

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\applications.py", line 289, in __call__
    await super().__call__(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\middleware\cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 20, in __call__
    raise e
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\routing.py", line 291, in app
    content = await serialize_response(
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\routing.py", line 179, in serialize_response
    return jsonable_encoder(response_content)
  File "D:\AI\WebAI-to-API\venv\lib\site-packages\fastapi\encoders.py", line 238, in jsonable_encoder
    raise ValueError(errors) from e
ValueError: [TypeError("'async_generator' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

johnd0e commented 10 months ago

As I see in code, streaming just is not yet implemented. E.g. https://github.com/Amm1rr/WebAI-to-API/blob/eafd56505a732f96d93a038fbbb036042b5cdca8/src/main.py#L25 For streaming here should be AsyncChatbot

Amm1rr / WebAI-to-API

Streaming is not working with examples. #11