Message Persistence for Streamed Responses - Githubissues

kevinthedang / discord-ollama

Discord Bot that utilizes Ollama to interact with any Large Language Models to talk with users and allow them to host/create their own models.

Creative Commons Attribution 4.0 International

42 stars 4 forks source link

Message Persistence for Streamed Responses #51

Closed kevinthedang closed 2 months ago

kevinthedang commented 2 months ago

Issue

When streaming a response from Ollama, it breaks down the message into "tokens" These tokens are separated on generation.
Refer to code snippet from ollama's documentation:

{
  "model": "llama2",
  "created_at": "2023-08-04T08:52:19.385406455-07:00",
  "message": {
    "role": "assistant",
    "content": "The",
    "images": null
  },
  "done": false
}

Solution

Implement a way to combine the stream or any other creative way.
- In terms of combining, we just need to iterate through the response and append them together as 1 string.