search

rendezqueue / rendezllama

CLI for llama.cpp with various commands to guide, edit, and regenerate tokens on the fly.
ISC License
10 stars 1 forks source link

feat(option): to apply a LoRA #18

Closed grencez closed 1 year ago

grencez commented 1 year ago

Not sure how useful this is without instruction/response format (#5), but it seems worth doing. It's also a good excuse to bump the llama.cpp version.

grencez commented 1 year ago

I tried the 13B SuperCOT LoRA and it seems to improve chat quality even without the instruction/response format. Though if you let it go on its own, it'll often try to add end-of-message strings like ### End of Transcript Continuation, presumably because its preferred format has lots of ### sections.

That said, the LoRA takes a while for me to load because of the higher-precision base model. My test machine's disk is kinda slow, but I guess that isn't the case for everyone.