This is a medical bot built using Llama2 and Sentence Transformers. The bot is powered by Langchain and Chainlit. The bot runs on a decent CPU machine with a minimum of 16GB of RAM.
As an LLM newbie I might be being too optimistic trying to run this with the Llama 2 quantized model _llama-2-7b-chat.Q4_KM.gguf on a cpu with only 8GB RAM nominal. The Chainlit page loads and after entering a question after a while it appears to timeout with the message "Could not reach the server". Should I increase the session_timeout parameter in config.toml or run some of the code in async mode?
Nice project and video!
As an LLM newbie I might be being too optimistic trying to run this with the Llama 2 quantized model _llama-2-7b-chat.Q4_KM.gguf on a cpu with only 8GB RAM nominal. The Chainlit page loads and after entering a question after a while it appears to timeout with the message "Could not reach the server". Should I increase the session_timeout parameter in config.toml or run some of the code in async mode?