search

henk717 / KoboldAI

KoboldAI is generative AI software optimized for fictional use, but capable of much more!
http://koboldai.com
GNU Affero General Public License v3.0
359 stars 130 forks source link

Feature request: rope scaling controls #442

Open BlairSadewitz opened 1 year ago

BlairSadewitz commented 1 year ago

Hello,

Today I tried dynamic rope scaling (factor 4) with the Holomax model, and it worked really well--used the entire 16k context without any obvious degradation in quality. Therefore, I'm wondering if you would modify the UI so users could take advantage of it without having to edit config.json. Perhaps you could tuck it away in the experimental UI at first. The user could turn on the experimental UI, then go to load a model. I propose something like this:

When the user selects a model to load, the code could check if it's a model with transformers support for that. If it's supported, there'd be a toggle for on/off. Then, it would check config.json. If it's specified as enabled there, it would default to those settings. Otherwise, it would default to "off" and "dynamic" for strategy and, let's say 2.0 for the factor (perhaps a slider, maybe with a text input box?). But if the "enabled" toggle is off, it doesn't display the other settings. The user would have to turn it on to see the settings. If it is specified in config.json, it would default to "on" with the settings defaulting to those specified in config.json.

If "experimental UI" isn't on, it would behave as it does now. What do you think?

henk717 commented 1 year ago

This is indeed planned, reasonably close to how you suggest it but without the experimental UI bit. The idea is adding a dropdown option for rope in the loading dialogue that is on Auto by default, in which case it doesn't do anything and lets HF handle it. If the user switches it to manual it would expose the manual configs you can pass to the model yourself.

BlairSadewitz commented 1 year ago

Yeah, that's essentially the logic I'm talking about. I approached it with the assumption that you'd want to make it "experimental" at first, but, if not, I surely won't complain. ;-) I went with an explicit "on/off" so that the UI would clearly reflect whether or not it actually IS specified in config.json by default or not--and the user wouldn't have to click on anything in order to see what the default settings are. The way you describe is arguably more elegant, but it's ultimately a matter of taste, I think.