search

p-e-w / arrows

A frontend for creative writing with LLMs
GNU Affero General Public License v3.0
106 stars 7 forks source link

Enabled configuration of remote APIs by moving model name parameter from main.ts to config.ts #8

Open fellowtechnologist opened 3 months ago

fellowtechnologist commented 3 months ago

Moved model name parameter from main.ts to config.ts.

This parameter is rarely needed for local APIs but needs to be configured for remote APIs that support completions, added note about format of model name and top-k value change sometimes required.

Tested successfully using APIs from together.ai and anyscale.com, failed test using groq.com API because completions aren't supported only chat/completions.

fellowtechnologist commented 2 months ago

@p-e-w hello, I'm afraid I'm not a dev and I've never actually suggested a pull request before, so apologies if my etiquette is off in any way.

I'm a big fan of Arrows but I found that I couldn't use various APIs until I'd found out where the model parameter was identified, when I found it and configured to identify the intended model, I could suddenly use any API which supported completions.

I tried out the above edits to simply move where the model parameter is identified onto the config page with the other parameters for convenience, as this is something I have to configure often, for example to use NeMo via the Openrouter API, because I have a very limited local llm capacity.

I've tried to submit this suggested change here in the hopes that it will enable more people to access and enjoy using Arrows but I'm afraid I don't really know what I'm doing with git so I don't know if this request has reached you or if there's a good reason not to make the update.

So this suggestion is offered with thanks from a user who's enjoying using arrows for their own writing practice. Thank you again for open sourcing Arrows, it's great!

p-e-w commented 2 months ago

This has been brought up several times before and I have laid out my reason for not having the model name as a parameter. Namely, with the actual OpenAI API, the model can not be freely changed because the newer OpenAI models don't support the text completion API.

I am currently testing several experimental changes on my local Arrows instance, and once I have integrated those, I plan to reconsider how the ability to select the model for alternative providers can best be reconciled with the constraints OpenAI places on that parameter.

Incidentally, the PARAMS object is spread into params, so it is already possible to override model without needing to change the application code.