Tensor parallelism is all you need. Run LLMs on weak devices or make powerful devices even more powerful by distributing the workload and dividing the RAM usage.
This PR introduces a simple cache in the api server. It speeds up the inference in a chat client like AnythingLLM by reusing already processed input tokens.
Also, this PR resolves the problem with added unwanted <|eot_id|> to the response.
This PR introduces a simple cache in the api server. It speeds up the inference in a chat client like AnythingLLM by reusing already processed input tokens.
Also, this PR resolves the problem with added unwanted
<|eot_id|>to the response.