Feature Request: Improve CPU processing speed for large contexts - Githubissues

ikawrakow / ik_llama.cpp

llama.cpp fork with additional SOTA quants and improved performance

MIT License

89 stars 6 forks source link

Feature Request: Improve CPU processing speed for large contexts #26

Open ikawrakow opened 2 months ago

ikawrakow commented 2 months ago

Prerequisites

[X] I am running the latest code. Mention the version if possible as well.
[X] I carefully followed the README.md.
[X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
[X] I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Recent open source / open weight models provide long context window, and hence it would be useful to improve CPU processing speed for large prompts.

Motivation

See #25

Possible Implementation

See #25