Ollama History Performance Issues

kevinthedang / discord-ollama

Discord Bot that utilizes Ollama to interact with any Large Language Models to talk with users and allow them to host/create their own models.

Creative Commons Attribution 4.0 International

45 stars 5 forks source link

Ollama History Performance Issues #7

Closed kevinthedang closed 5 months ago

kevinthedang commented 5 months ago

Issue

Response times are linear in accordance with when a message has been sent, this will be a problem as if (for example) 5 people want to query the boy at the same time, it runs the queries linearly which will be inconvenient.

Notes

There probably won't be that many users using it at the same time so there will be a static capacity on threads
Future feature would be to dynamically check cores on a system and figure out how many threads to allow Ollama to run on and set in the settings (Util feature).

JT2M0L3Y commented 5 months ago

Conversations can be piped into individual threads via a slash command added by @kevinthedang.

kevinthedang commented 5 months ago

Sorry I chose a poor name for this. This should be about how Ollama performs as more messages appear.

Update

Figure out some way to increase performance of models with message history
Redis can be something we can use for history...

kevinthedang commented 5 months ago

Update

To increase performance as of now, we can use a docker image.
Refer to the following:
Also worth looking into two repos:
- ollama_proxy_server
- ollama_docker_compose

kevinthedang commented 5 months ago

For now, we will not worry about performance issues as it is mainly a hardware problem when it comes down to it. This will be a response time issue for when multiple users access the bot at the same time.