search

p-e-w / arrows

A frontend for creative writing with LLMs
GNU Affero General Public License v3.0
106 stars 7 forks source link

Model should be configurable #2

Closed StableLlama closed 4 months ago

StableLlama commented 4 months ago

Currently the model gpt-3.5-turbo-instruct is hardly configured in the sources. This should be an option for the config file.

And best would be to ask the server about the available models and list them, so that it's easier to select one available model.

This feature is important when you don't run the AI server with just one model locally but instead you have a campus server that serves multiple models at the same time.

p-e-w commented 4 months ago

gpt-3.5-turbo-instruct is hardcoded because that's the only modern OpenAI model that supports the text completion (rather than chat completion) endpoint that Arrows relies on. Local loaders ignore that parameter, as documented in the source code.

A configuration UI such as a model list isn't going to happen. The proper place to select the model is in the loader.

This feature is important when you don't run the AI server with just one model locally but instead you have a campus server that serves multiple models at the same time.

The OpenAI API, which Arrows uses, does not support such a configuration. The model parameter is not arbitrary but an enumeration of OpenAI models. Local OpenAI-compatible providers don't use that parameter but (if anything) only support globally switching the loaded model (for all users), not the multi-model server you are describing.

StableLlama commented 4 months ago

gpt-3.5-turbo-instruct is hardcoded because that's the only modern OpenAI model that supports the text completion (rather than chat completion) endpoint that Arrows relies on. Local loaders ignore that parameter, as documented in the source code.

We have so many more models than those from OpenAI. And changing the source code I could easily switch between them.

The OpenAI API, which Arrows uses, does not support such a configuration. The model parameter is not arbitrary but an enumeration of OpenAI models. Local OpenAI-compatible providers don't use that parameter but (if anything) only support globally switching the loaded model (for all users), not the multi-model server you are describing.

Oh, it does. Look e.g. at https://helmholtz-blablador.fz-juelich.de:8000/docs where you can get this list of models: image

p-e-w commented 4 months ago

We have so many more models than those from OpenAI. And changing the source code I could easily switch between them.

No need to change the source. Note that the PARAMS config constant is spread into params, so you can just specify a different value for model inside PARAMS and it will override the default.

But I will not add a separate configuration option for this, because most loaders ignore that value, and choosing a value that is not an OpenAI model name will result in an error with the official OpenAI API.