Add "context swap" functions to session and add "decoded_tokens" to snapshot read/write

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models

https://docs.rs/llm/latest/llm/

Apache License 2.0

6.06k stars 350 forks source link

Add "context swap" functions to session and add "decoded_tokens" to snapshot read/write #424

Closed Andreybest closed 7 months ago

Andreybest commented 9 months ago

Good day! Thank you for this awesome lib. I've added feature for "context swap" similar on how is one made in llama.cpp cli. And a small fix to include "decoded_tokens" in snapshots Im not sure if my code can go with all rust coder standards (coding for more than a month in rust). Please let me know if something needs a change.

philpax commented 7 months ago

Hi! Sorry for the very late response - I'm catching up on pending PRs by merging them into a develop branch so that they can be unified without impacting main.

Great work - the only changes I'd make are to the interface, but I figured I'd merge your code as-is because I need to redesign the overall interface anyway. Much appreciated 🙂