search

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models
https://docs.rs/llm/latest/llm/
Apache License 2.0
6.08k stars 362 forks source link

Make a Chat like Application #52

Closed hhamud closed 1 year ago

hhamud commented 1 year ago

Creating an issue here following discussion in the discord chat.

erlend-sh commented 1 year ago

This will be solved by #49

hhamud commented 1 year ago

This will be solved by #49

I specifically want a lightweight cli version with a sliding window for context building. Maybe he could rip that part out of his PR if he's built that portion and integrate that into the library? @karelnagel

karelnagel commented 1 year ago

This will be solved by #49

I specifically want a lightweight cli version with a sliding window for context building. Maybe he could rip that part out of his PR if he's built that portion and integrate that into the library? @karelnagel

In my app I just keep the model running and so it has all the previous context for that session. I think it should be quite easy to implement that same logic into the cli as well with some kind of flag. I basically just stole that logic from this PR https://github.com/setzer22/llama-rs/pull/37 .😀

setzer22 commented 1 year ago

As mentioned on the discord conversation, the real challenge here is extending the context window beyond the current cap of 2048 tokens. But in the meantime, a chat application with a 2048 message hard cap would be interesting to have.

I've tried several experiments to achieve a sliding context window, but nothing that gave good results so far. So for now, I'm trying to build a bit more understanding of the underlying transformer architecture before I attempt this again.

Other ways of achieving a larger context window would be training a larger model from scratch (unfeasible), or fine tuning an existing model to work with a larger context window. I'm not entirely sure about the effectiveness of the latter strategy.

philpax commented 1 year ago

This is related to #53 and #76.

philpax commented 1 year ago

This now exists in the CLI, but it doesn't have a sliding window. Closing this issue and moving discussion to #77.